Controlling representations of virtual objects in a computer-generated reality environment

ABSTRACT

In accordance with some embodiments, an exemplary process for controlling representations of virtual objects based on respective user contexts that each correspond to different respective locations in a computer-generated reality (CGR) environment is described.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of PCT/US2020/048833, entitled“CONTROLLING REPRESENTATIONS OF VIRTUAL OBJECTS IN A COMPUTER-GENERATEDREALITY ENVIRONMENT,” filed Aug. 31, 2020, which claims priority to U.S.Provisional Patent Application Ser. No. 63/058,217, entitled“CONTROLLING REPRESENTATIONS OF VIRTUAL OBJECTS IN A COMPUTER-GENERATEDREALITY ENVIRONMENT,” filed on Jul. 29, 2020; and U.S. ProvisionalPatent Application Ser. No. 62/907,216, entitled “CONTROLLINGREPRESENTATIONS OF VIRTUAL OBJECTS BASED ON USE CONTEXTS OF LOCATIONS INA COMPUTER-GENERATED REALITY ENVIRONMENT,” filed on Sep. 27, 2019. Thecontents of the aforementioned applications are hereby incorporated byreference in their entireties.

FIELD

The present disclosure relates generally to computer-generated realityenvironments, and more specifically to controlling representations ofvirtual objects in a computer-generated reality environment.

BACKGROUND

Computer-generated reality (CGR) environments are environments where atleast some objects displayed for a user's viewing are generated by acomputer. In some uses, a user may interact with virtual objects. Theuser may move the virtual objects to different locations within thecomputer-generated reality environment. However, there is a lack ofrobust functionality for controlling how the virtual objects arerepresented at the different locations within the computer-generatedreality environments. In addition, there is a lack of functionality forcontrolling representations of virtual objects in the computer-generatedenvironments based on characteristics of an input mechanism that may beused by the users to interact with the virtual objects.

Furthermore, the virtual objects may be represented as two-dimensionalobjects and/or three-dimensional objects within the computer-generatedreality environments. However, there is a lack of robust functionalityfor controlling the concurrent display of the representations of thevirtual objects.

BRIEF SUMMARY

In accordance with some embodiments, a method comprises: displaying, viaa display of an electronic device, a first representation of a virtualobject at a first location within a CGR environment, wherein in thefirst location corresponds to a first use context of a plurality of usecontexts; receiving a request to move the first representation, withinthe CGR environment, to a second location that is different from thefirst location; and in response to receiving the request: in accordancewith a determination that the second location corresponds to a seconduse context of the plurality of use contexts, displaying, via thedisplay of the electronic device, at the second location, a secondrepresentation of the virtual object based on the second use context,wherein the second representation is different from the firstrepresentation; and in accordance with a determination that the secondlocation corresponds to a third use context of the plurality of usecontexts, displaying, via the display of the electronic device, at thesecond location, a third representation of the virtual object based onthe third use context, wherein the third representation is differentfrom the first representation and the second representation.

In accordance with some embodiments, a system includes: a display; oneor more processors; and memory storing one or more programs configuredto be executed by the one or more processors. The one or more programsincludes instructions for: displaying, via the display, a firstrepresentation of a virtual object at a first location within a CGRenvironment, wherein the first location corresponds to a first usecontext of a plurality of use contexts; receiving a request to move thefirst representation, within the CGR environment, to a second locationthat is different from the first location; and in response to receivingthe request: in accordance with a determination that the second locationcorresponds to a second use context of the plurality of use contexts,displaying, via the display, at the second location, a secondrepresentation of the virtual object based on the second use context,wherein the second representation is different from the firstrepresentation; and in accordance with a determination that the secondlocation corresponds to a third use context of the plurality of usecontexts, displaying, via the display, at the second location, a thirdrepresentation of the virtual object based on the third use context,wherein the third representation is different from the firstrepresentation and the second representation.

In accordance with some embodiments, a non-transitory computer-readablestorage medium stores one or more programs configured to be executed byone or more processors of a system having a display, the one or moreprograms including instructions for: displaying, via the display, afirst representation of a virtual object at a first location within aCGR environment, wherein the first location corresponds to a first usecontext of a plurality of use contexts; receiving a request to move thefirst representation, within the CGR environment, to a second locationthat is different from the first location; and in response to receivingthe request: in accordance with a determination that the second locationcorresponds to a second use context of the plurality of use contexts,displaying, via the display, at the second location, a secondrepresentation of the virtual object based on the second use context,wherein the second representation is different from the firstrepresentation; and in accordance with a determination that the secondlocation corresponds to a third use context of the plurality of usecontexts, displaying, via the display, at the second location, a thirdrepresentation of the virtual object based on the third use context,wherein the third representation is different from the firstrepresentation and the second representation.

In accordance with some embodiments, a transitory computer-readablestorage medium stores one or more programs configured to be executed byone or more processors of a system having a display, the one or moreprograms including instructions for: displaying, via the display, afirst representation of a virtual object at a first location within aCGR environment, wherein the first location corresponds to a first usecontext of a plurality of use contexts; receiving a request to move thefirst representation, within the CGR environment, to a second locationthat is different from the first location; and in response to receivingthe request: in accordance with a determination that the second locationcorresponds to a second use context of the plurality of use contexts,displaying, via the display, at the second location, a secondrepresentation of the virtual object based on the second use context,wherein the second representation is different from the firstrepresentation; and in accordance with a determination that the secondlocation corresponds to a third use context of the plurality of usecontexts, displaying, via the display, at the second location, a thirdrepresentation of the virtual object based on the third use context,wherein the third representation is different from the firstrepresentation and the second representation.

In accordance with some embodiments, a system includes: a display; meansfor displaying, via the display, a first representation of a virtualobject at a first location within a CGR environment, wherein the firstlocation corresponds to a first use context of a plurality of usecontexts; means for receiving a request to move the firstrepresentation, within the CGR environment, to a second location that isdifferent from the first location; and means, responsive to receivingthe request, for: in accordance with a determination that the secondlocation corresponds to a second use context of the plurality of usecontexts, displaying, via the display, at the second location, a secondrepresentation of the virtual object based on the second use context,wherein the second representation is different from the firstrepresentation; and in accordance with a determination that the secondlocation corresponds to a third use context of the plurality of usecontexts, displaying, via the display, at the second location, a thirdrepresentation of the virtual object based on the third use context,wherein the third representation is different from the firstrepresentation and the second representation.

In accordance with some embodiments, a method comprises: displaying, viaa display of an electronic device, a two-dimensional (2D) representationof a virtual object at a first location of a CGR environment; receivinga request to concurrently display a three-dimensional (3D)representation of the virtual object with the 2D representation; and inresponse to the request, concurrently displaying, via the display of thewearable electronic device, the 2D representation at the first locationand the 3D representation at a second location of the CGR environment,wherein the second location is different from the first location.

In accordance with some embodiments, a system includes: a display; oneor more processors; and memory storing one or more programs configuredto be executed by the one or more processors. The one or more programsincludes instructions for: displaying, via the display, a 2Drepresentation of a virtual object at a first location of a CGRenvironment; receiving a request to concurrently display a 3Drepresentation of the virtual object with the 2D representation; and inresponse to the request, concurrently displaying, via the display, the2D representation at the first location and the 3D representation at asecond location of the CGR environment, wherein the second location isdifferent from the first location.

In accordance with some embodiments, a non-transitory computer-readablestorage medium stores one or more programs configured to be executed byone or more processors of a system having a display, the one or moreprograms including instructions for: displaying, via the display, a 2Drepresentation of a virtual object at a first location of a CGRenvironment; receiving a request to concurrently display a 3Drepresentation of the virtual object with the 2D representation; and inresponse to the request, concurrently displaying, via the display, the2D representation at the first location and the 3D representation at asecond location of the CGR environment, wherein the second location isdifferent from the first location.

In accordance with some embodiments, a transitory computer-readablestorage medium stores one or more programs configured to be executed byone or more processors of a system having a display, the one or moreprograms including instructions for: displaying, via the display, a 2Drepresentation of a virtual object at a first location of a CGRenvironment; receiving a request to concurrently display a 3Drepresentation of the virtual object with the 2D representation; and inresponse to the request, concurrently displaying, via the display, the2D representation at the first location and the 3D representation at asecond location of the CGR environment, wherein the second location isdifferent from the first location.

In accordance with some embodiments, a system includes: a display; meansfor displaying, via the display, a 2D representation of a virtual objectat a first location of a CGR environment; means for receiving a requestto concurrently display a 3D representation of the virtual object withthe 2D representation; and in response to the request, means forconcurrently displaying, via the display, the 2D representation at thefirst location and the 3D representation at a second location of the CGRenvironment, wherein the second location is different from the firstlocation.

In accordance with some embodiments, a method comprises: displaying, viaa display of a wearable electronic device, a first representation of avirtual object within a CGR environment; and in response to a detectedmovement of an input mechanism: in accordance with a determination thata current location of the input mechanism is within a predetermineddistance from the first representation of the virtual object,displaying, via the display of the wearable electronic device, a secondrepresentation of the virtual object within the CGR environment, whereinthe second representation is different from the first representation;and in accordance with a determination that the current location of theinput mechanism is not within the predetermined distance from the firstrepresentation of the virtual object, maintaining display of the firstrepresentation without displaying the second representation.

In accordance with some embodiments, a system includes: a display; oneor more processors; and memory storing one or more programs configuredto be executed by the one or more processors. The one or more programsincludes instructions for: displaying, via the display, a firstrepresentation of a virtual object within a CGR environment; and inresponse to a detected movement of an input mechanism: in accordancewith a determination that a current location of the input mechanism iswithin a predetermined distance from the first representation of thevirtual object, displaying, via the display, a second representation ofthe virtual object within the CGR environment, wherein the secondrepresentation is different from the first representation; and inaccordance with a determination that the current location of the inputmechanism is not within the predetermined distance from the firstrepresentation of the virtual object, maintaining display of the firstrepresentation without displaying the second representation.

In accordance with some embodiments, a non-transitory computer-readablestorage medium stores one or more programs configured to be executed byone or more processors of a system having a display, the one or moreprograms including instructions for: displaying, via the display, afirst representation of a virtual object within a CGR environment; andin response to a detected movement of an input mechanism: in accordancewith a determination that a current location of the input mechanism iswithin a predetermined distance from the first representation of thevirtual object, displaying, via the display, a second representation ofthe virtual object within the CGR environment, wherein the secondrepresentation is different from the first representation; and inaccordance with a determination that the current location of the inputmechanism is not within the predetermined distance from the firstrepresentation of the virtual object, maintaining display of the firstrepresentation without displaying the second representation.

In accordance with some embodiments, a transitory computer-readablestorage medium stores one or more programs configured to be executed byone or more processors of a system having a display, the one or moreprograms including instructions for: displaying, via the display, afirst representation of a virtual object within a CGR environment; andin response to a detected movement of an input mechanism: in accordancewith a determination that a current location of the input mechanism iswithin a predetermined distance from the first representation of thevirtual object, displaying, via the display, a second representation ofthe virtual object within the CGR environment, wherein the secondrepresentation is different from the first representation; and inaccordance with a determination that the current location of the inputmechanism is not within the predetermined distance from the firstrepresentation of the virtual object, maintaining display of the firstrepresentation without displaying the second representation.

In accordance with some embodiments, a system includes: a display; meansfor displaying, via the display, a first representation of a virtualobject within a CGR environment; and in response to a detected movementof an input mechanism: means for, in accordance with a determinationthat a current location of the input mechanism is within a predetermineddistance from the first representation of the virtual object,displaying, via the display, a second representation of the virtualobject within the CGR environment, wherein the second representation isdifferent from the first representation; and means for, in accordancewith a determination that the current location of the input mechanism isnot within the predetermined distance from the first representation ofthe virtual object, maintaining display of the first representationwithout displaying the second representation.

Executable instructions for performing these functions are, optionally,included in a non-transitory computer-readable storage medium or othercomputer program product configured for execution by one or moreprocessors. Executable instructions for performing these functions are,optionally, included in a transitory computer-readable storage medium orother computer program product configured for execution by one or moreprocessors.

BRIEF DESCRIPTION OF THE FIGURES

In the following description, reference is made to the accompanyingfigures which form a part thereof, and which illustrate several examplesof the present disclosure. It is understood that other examples may beutilized and structural and operational changes may be made withoutdeparting from the scope of the present disclosure. The use of the samereference symbols in different figures indicates similar or identicalitems.

FIGS. 1A-1B depict exemplary systems for use in variouscomputer-generated reality technologies.

FIGS. 2A-2E depict exemplary techniques for controlling a representationof a virtual object based on use contexts at different locations withina CGR environment in accordance with aspects of the present disclosure.

FIGS. 3A-3C depict another example of techniques for controlling arepresentation of a virtual object based on use contexts at differentlocations within a CGR environment in accordance with aspects of thepresent disclosure.

FIGS. 4A-4C depict yet another example of techniques for controlling arepresentation of a virtual object based on use contexts at differentlocations within a CGR environment in accordance with aspects of thepresent disclosure.

FIG. 5 is a flow diagram illustrating a method executed to implementaspects of the present disclosure.

FIGS. 6A-6C depict exemplary techniques for controlling a concurrentdisplay of representations of a virtual object within a CGR environmentin accordance with aspects of the present disclosure.

FIG. 7 is a flow diagram illustrating a method executed to implementaspects of the present disclosure.

FIGS. 8A-8B depict exemplary techniques for controlling a representationof a virtual object within a CGR environment based on characteristics ofan input mechanism in accordance with aspects of the present disclosure.

FIGS. 9A-9B depict another example of techniques for controlling arepresentation of a virtual object within a CGR environment based oncharacteristics of an input mechanism in accordance with aspects of thepresent disclosure.

FIGS. 10A-10B depict yet another example of techniques for controlling arepresentation of a virtual object within a CGR environment based oncharacteristics of an input mechanism in accordance with aspects of thepresent disclosure.

FIGS. 11A-11B depict still another example of techniques for controllinga representation of a virtual object within a CGR environment based oncharacteristics of an input mechanism in accordance with aspects of thepresent disclosure.

FIGS. 12A-12B depict another example of techniques for controlling arepresentation of a virtual object within a CGR environment based oncharacteristics of an input mechanism in accordance with aspects of thepresent disclosure.

FIG. 13 is a flow diagram illustrating a method executed to implementaspects of the present disclosure.

DESCRIPTION

Various examples of electronic systems and techniques for using suchsystems in relation to various CGR technologies are described.

A physical environment (or real environment) refers to a physical worldthat people can sense and/or interact with without aid of electronicsystems. Physical environments, such as a physical park, includephysical articles (or physical objects or real objects), such asphysical trees, physical buildings, and physical people. People candirectly sense and/or interact with the physical environment, such asthrough sight, touch, hearing, taste, and smell.

In contrast, a CGR environment refers to a wholly or partially simulatedenvironment that people sense and/or interact with via an electronicsystem. In CGR, a subset of a person's physical motions, orrepresentations thereof, are tracked, and, in response, one or morecharacteristics of one or more virtual objects simulated in the CGRenvironment are adjusted in a manner that comports with at least one lawof physics. For example, a CGR system may detect a person's head turningand, in response, adjust graphical content and an acoustic fieldpresented to the person in a manner similar to how such views and soundswould change in a physical environment. In some situations (e.g., foraccessibility reasons), adjustments to characteristic(s) of virtualobject(s) in a CGR environment may be made in response torepresentations of physical motions (e.g., vocal commands).

A person may sense and/or interact with a CGR object using any one oftheir senses, including sight, sound, touch, taste, and smell. Forexample, a person may sense and/or interact with audio objects thatcreate a three-dimensional (3D) or spatial audio environment thatprovides the perception of point audio sources in 3D space. In anotherexample, audio objects may enable audio transparency, which selectivelyincorporates ambient sounds from the physical environment with orwithout computer-generated audio. In some CGR environments, a person maysense and/or interact only with audio objects.

Examples of CGR include virtual reality and mixed reality.

A virtual reality (VR) environment (or virtual environment) refers to asimulated environment that is designed to be based entirely oncomputer-generated sensory inputs for one or more senses. A VRenvironment comprises a plurality of virtual objects with which a personmay sense and/or interact. For example, computer-generated imagery oftrees, buildings, and avatars representing people are examples ofvirtual objects. A person may sense and/or interact with virtual objectsin the VR environment through a simulation of the person's presencewithin the computer-generated environment, and/or through a simulationof a subset of the person's physical movements within thecomputer-generated environment.

In contrast to a VR environment, which is designed to be based entirelyon computer-generated sensory inputs, a mixed reality (MR) environmentrefers to a simulated environment that is designed to incorporatesensory inputs from the physical environment, or a representationthereof, in addition to including computer-generated sensory inputs(e.g., virtual objects). On a virtuality continuum, an MR environment isanywhere between, but not including, a wholly physical environment atone end and a VR environment at the other end.

In some MR environments, computer-generated sensory inputs may respondto changes in sensory inputs from the physical environment. Also, someelectronic systems for presenting an MR environment may track locationand/or orientation with respect to the physical environment to enablevirtual objects to interact with real objects (that is, physicalarticles from the physical environment or representations thereof). Forexample, a system may account for movements so that a virtual treeappears stationary with respect to the physical ground.

Examples of MR Include Augmented Reality and Augmented Virtuality

An augmented reality (AR) environment refers to a simulated environmentin which one or more virtual objects are superimposed over a physicalenvironment, or a representation thereof. For example, an electronicsystem for presenting an AR environment may have a transparent ortranslucent display through which a person may directly view thephysical environment. The system may be configured to present virtualobjects on the transparent or translucent display, so that a person,using the system, perceives the virtual objects superimposed over thephysical environment. Alternatively, a system may have an opaque displayand one or more imaging sensors that capture images or video of thephysical environment, which are representations of the physicalenvironment. The system composites the images or video with virtualobjects, and presents the composition on the opaque display. A person,using the system, indirectly views the physical environment by way ofthe images or video of the physical environment, and perceives thevirtual objects superimposed over the physical environment. As usedherein, a video of the physical environment shown on an opaque displayis called “pass-through video,” meaning a system uses one or more imagesensor(s) to capture images of the physical environment, and uses thoseimages in presenting the AR environment on the opaque display. Furtheralternatively, a system may have a projection system that projectsvirtual objects into the physical environment, for example, as ahologram or on a physical surface, so that a person, using the system,perceives the virtual objects superimposed over the physicalenvironment.

An AR environment also refers to a simulated environment in which arepresentation of a physical environment is transformed bycomputer-generated sensory information. For example, in providingpass-through video, a system may transform one or more sensor images toimpose a select perspective (e.g., viewpoint) different than theperspective captured by the imaging sensors. As another example, arepresentation of a physical environment may be transformed bygraphically modifying (e.g., enlarging) portions thereof, such that themodified portion may be representative but not photorealistic versionsof the originally captured images. As a further example, arepresentation of a physical environment may be transformed bygraphically eliminating or obfuscating portions thereof.

An augmented virtuality (AV) environment refers to a simulatedenvironment in which a virtual or computer generated environmentincorporates one or more sensory inputs from the physical environment.The sensory inputs may be representations of one or more characteristicsof the physical environment. For example, an AV park may have virtualtrees and virtual buildings, but people with faces photorealisticallyreproduced from images taken of physical people. As another example, avirtual object may adopt a shape or color of a physical article imagedby one or more imaging sensors. As a further example, a virtual objectmay adopt shadows consistent with the position of the sun in thephysical environment.

There are many different types of electronic systems that enable aperson to sense and/or interact with various CGR environments. Examplesinclude head mounted systems, projection-based systems, heads-updisplays (HUDs), vehicle windshields having integrated displaycapability, windows having integrated display capability, displaysformed as lenses designed to be placed on a person's eyes (e.g., similarto contact lenses), headphones/earphones, speaker arrays, input systems(e.g., wearable or handheld controllers with or without hapticfeedback), smartphones, tablets, and desktop/laptop computers. A headmounted system may have one or more speaker(s) and an integrated opaquedisplay. Alternatively, a head mounted system may be configured toaccept an external opaque display (e.g., a smartphone). The head mountedsystem may incorporate one or more imaging sensors to capture images orvideo of the physical environment, and/or one or more microphones tocapture audio of the physical environment. Rather than an opaquedisplay, a head mounted system may have a transparent or translucentdisplay. The transparent or translucent display may have a mediumthrough which light representative of images is directed to a person'seyes. The display may utilize digital light projection, OLEDs, LEDs,uLEDs, liquid crystal on silicon, laser scanning light source, or anycombination of these technologies. The medium may be an opticalwaveguide, a hologram medium, an optical combiner, an optical reflector,or any combination thereof. In one example, the transparent ortranslucent display may be configured to become opaque selectively.Projection-based systems may employ retinal projection technology thatprojects graphical images onto a person's retina. Projection systemsalso may be configured to project virtual objects into the physicalenvironment, for example, as a hologram or on a physical surface.

FIG. 1A and FIG. 1B depict exemplary system 100 for use in various CGRtechnologies.

In some examples, as illustrated in FIG. 1A, system 100 includes device100 a. Device 100 a includes various components, such as processor(s)102, RF circuitry(ies) 104, memory(ies) 106, image sensor(s) 108,orientation sensor(s) 110, microphone(s) 112, location sensor(s) 116,speaker(s) 118, display(s) 120, and touch-sensitive surface(s) 122.These components optionally communicate over communication bus(es) 150of device 100 a.

In some examples, elements of system 100 are implemented in a basestation device (e.g., a computing device, such as a remote server,mobile device, or laptop) and other elements of the system 100 areimplemented in a head-mounted display (HMD) device designed to be wornby the user, where the HMD device is in communication with the basestation device. In some examples, device 100 a is implemented in a basestation device or a HMD device.

As illustrated in FIG. 1B, in some examples, system 100 includes two (ormore) devices in communication, such as through a wired connection or awireless connection. First device 100 b (e.g., a base station device)includes processor(s) 102, RF circuitry(ies) 104, and memory(ies) 106.These components optionally communicate over communication bus(es) 150of device 100 b. Second device 100 c (e.g., a HMD) includes variouscomponents, such as processor(s) 102, RF circuitry(ies) 104, memory(ies)106, image sensor(s) 108, orientation sensor(s) 110, microphone(s) 112,location sensor(s) 116, speaker(s) 118, display(s) 120, andtouch-sensitive surface(s) 122. These components optionally communicateover communication bus(es) 150 of device 100 c.

In some examples, system 100 is a mobile device. In some examples,system 100 is an HMD device. In some examples, system 100 is a wearableHUD device.

System 100 includes processor(s) 102 and memory(ies) 106. Processor(s)102 include one or more general processors, one or more graphicsprocessors, and/or one or more digital signal processors. In someexamples, memory(ies) 106 are one or more non-transitorycomputer-readable storage mediums (e.g., flash memory, random accessmemory) that store computer-readable instructions configured to beexecuted by processor(s) 102 to perform the techniques described below.

System 100 includes RF circuitry(ies) 104. RF circuitry(ies) 104optionally include circuitry for communicating with electronic devices,networks, such as the Internet, intranets, and/or a wireless network,such as cellular networks and wireless local area networks (LANs). RFcircuitry(ies) 104 optionally includes circuitry for communicating usingnear-field communication and/or short-range communication, such asBluetooth®.

System 100 includes display(s) 120. In some examples, display(s) 120include a first display (e.g., a left eye display panel) and a seconddisplay (e.g., a right eye display panel), each display for displayingimages to a respective eye of the user. Corresponding images aresimultaneously displayed on the first display and the second display.Optionally, the corresponding images include the same virtual objectsand/or representations of the same physical objects from differentviewpoints, resulting in a parallax effect that provides a user with theillusion of depth of the objects on the displays. In some examples,display(s) 120 include a single display. Corresponding images aresimultaneously displayed on a first area and a second area of the singledisplay for each eye of the user. Optionally, the corresponding imagesinclude the same virtual objects and/or representations of the samephysical objects from different viewpoints, resulting in a parallaxeffect that provides a user with the illusion of depth of the objects onthe single display.

In some examples, system 100 includes touch-sensitive surface(s) 122 forreceiving user inputs, such as tap inputs and swipe inputs. In someexamples, display(s) 120 and touch-sensitive surface(s) 122 formtouch-sensitive display(s).

System 100 includes image sensor(s) 108. Image sensors(s) 108 optionallyinclude one or more visible light image sensor, such as charged coupleddevice (CCD) sensors, and/or complementary metal-oxide-semiconductor(CMOS) sensors operable to obtain images of physical objects from thereal environment. Image sensor(s) also optionally include one or moreinfrared (IR) sensor(s), such as a passive IR sensor or an active IRsensor, for detecting infrared light from the real environment. Forexample, an active IR sensor includes an IR emitter, such as an IR dotemitter, for emitting infrared light into the real environment. Imagesensor(s) 108 also optionally include one or more event camera(s)configured to capture movement of physical objects in the realenvironment. Image sensor(s) 108 also optionally include one or moredepth sensor(s) configured to detect the distance of physical objectsfrom system 100. In some examples, system 100 uses CCD sensors, eventcameras, and depth sensors in combination to detect the physicalenvironment around system 100. In some examples, image sensor(s) 108include a first image sensor and a second image sensor. The first imagesensor and the second image sensor are optionally configured to captureimages of physical objects in the real environment from two distinctperspectives. In some examples, system 100 uses image sensor(s) 108 toreceive user inputs, such as hand gestures. In some examples, system 100uses image sensor(s) 108 to detect the position and orientation ofsystem 100 and/or display(s) 120 in the real environment. For example,system 100 uses image sensor(s) 108 to track the position andorientation of display(s) 120 relative to one or more fixed objects inthe real environment.

In some examples, system 100 includes microphones(s) 112. System 100uses microphone(s) 112 to detect sound from the user and/or the realenvironment of the user. In some examples, microphone(s) 112 includes anarray of microphones (including a plurality of microphones) thatoptionally operate in tandem, such as to identify ambient noise or tolocate the source of sound in space of the real environment.

System 100 includes orientation sensor(s) 110 for detecting orientationand/or movement of system 100 and/or display(s) 120. For example, system100 uses orientation sensor(s) 110 to track changes in the positionand/or orientation of system 100 and/or display(s) 120, such as withrespect to physical objects in the real environment. Orientationsensor(s) 110 optionally include one or more gyroscopes and/or one ormore accelerometers.

Various aspects of the present disclosure are directed to systems andtechniques that provide functionality for controlling representations ofvirtual objects within a CGR environment.

In particular, aspects of the present disclosure are directed to systemsand techniques that provide functionality for controlling arepresentation of a virtual object based on a use context associatedwith a location of the virtual object within the CGR environment. Thesystems and techniques described herein allow for a representation of avirtual object to be adapted to the particular use context associatedwith the location within the CGR environment.

FIGS. 2A-2E illustrate exemplary techniques for controlling arepresentation of a virtual object of a CGR environment based on a usecontext associated with a location of the virtual object within the CGRenvironment in accordance with aspects of the present disclosure. Inparticular, FIG. 2A illustrates user 202 and electronic device 200. Insome embodiments, electronic device 200 may be a wearable electronicdevice (e.g., an HMD). Examples of a wearable electronic device aredescribed herein, such as with respect to electronic device 100 adescribed above with reference to FIGS. 1A and 1B.

As shown in FIG. 2A, user 202 wears electronic device 200, which isconfigured to enable user 202 to perceive CGR environment 290. Asdescribed above, CGR environment 290 may include physical objects, orrepresentations thereof, and virtual objects, with virtual objectssuperimposed upon the physical objects (e.g., in AR implementations), orphysical objects superimposed upon the virtual objects (e.g., in AVimplementations) to present a coherent CGR environment to user 202. Insome embodiments, CGR environment 290 may be a wholly virtualenvironment (e.g., in VR implementations), in which every object withinCGR environment 290 is a virtual object. Whether entirely or partiallyvirtual implementations, in the example illustrated in FIG. 2A, virtualobject 210 may be a representation of a presentation application (e.g.,an application configured to facilitate multimedia presentations) andmay be presented to user 202 within CGR environment 290.

In embodiments, virtual object 210 may be located at any location withinCGR environment 290. In the particular example illustrated in FIGS.2A-2E, CGR environment 290 may include at least location 220, 222, 224,226, and 228. As will be appreciated, these locations are described forillustration purposes and not intended to be limiting in any way. Thatis, any other location within CGR environment 290 may be applicable tothe features and functionalities described herein.

In aspects, location 220 may correspond to a location on arepresentation of an electronic device within CGR environment 290. Forexample, location 220 may correspond to a location (e.g., a display, ascreen, a surface or case of an electronic device) on display 240.Display 240 may be, for example, a display of a computer, laptop,tablet, phone, display, projector display, etc. Display 240 may be anactual physical device (e.g., a physical object) or may be a virtualrepresentation of a display (e.g., a virtual object) within CGRenvironment 290.

Location 222 may correspond to a location on a vertical plane of CGRenvironment 290 (e.g., a predominantly vertical plane such as astructure that is a vertical plane, a wall, a surface that correspondsto a wall-like structure such as a side of building, bedroom wall,fence, a vertical or auxiliary vertical plane, etc.). In the particularexample illustrated in FIG. 2A, location 222 corresponds to a locationon a wall of CGR environment 290.

Location 224, and/or location 228, may correspond to a location on ahorizontal plane of CGR environment 290 (e.g., a predominantlyhorizontal plane such as a structure that is a horizontal plane, adesktop, table, countertop, shelf, floor, an elevated horizontal planesuch as a horizontal plane that is above another horizontal plane withinthe CGR environment, a horizontal plane that is not elevated, etc.). Inthe particular example illustrated in FIG. 2A, locations 224 and 228correspond to locations on desktop 242, which may be a physical orvirtual object.

Location 226 may correspond to a location on a horizontal plane of CGRenvironment 290, but of a different type than locations 224 and/or 228.For example, location 226 may be a location on a predominantlyhorizontal plane such as a structure that is a horizontal plane, afloor, a sidewalk, grass, lawn, a surface that one or more people arestanding on, a non-elevated horizontal plane such as a horizontal planethat is below another horizontal plane within the CGR, etc. In theparticular example illustrated in FIG. 2A, location 226 corresponds to alocations on the floor of CGR environment 290.

As shown in FIG. 2A, virtual object 210 may be displayed at location 220(e.g., by electronic device 200). In some embodiments, a location withinCGR environment 290 (e.g., location 220) may be associated with orotherwise correspond to at least one use context of a plurality of usecontexts. In embodiments, a use context may be related to a type ofsurface (e.g., a desk, a wall, a computer screen, a floor, etc.) or thetype of material of the surface (e.g., sand, grass, concrete, carpet,etc.) that the virtual object will be placed on, and/or may be relatedto a manner in which the virtual object will be used (e.g., manipulated,interacted with) or displayed (e.g., presented) in the CGR environment.

In aspects, location 220 may be associated with a first use context. Forexample, as described above, location 220 may be a location on display240. Display 240 may be a representation of an electronic device. Inthis case, the first use context associated with location 220 may be thetype of surface or object of location 220, which is an electronicdevice. Thus, in this case, the first use context may be satisfied whena determination is made that location 220 is a location on arepresentation of an electronic device. In other embodiments, the firstuse context associated with location 220 may be the manner in whichvirtual object 210 will be used when in location 220. For example, itmay be determined that, at location 220, which is an electronic device,virtual object 220 will be used as an application for multimediapresentations on display 240. In this case, it may be determined thatvirtual object is to be represented as a two-dimensional (2D) windowbased on the manner in which virtual object will be used.

It is noted that, as used herein, a representation of a virtual objectmay include the content, size, functionality, user interface objects,form, shape, design, graphical presentation of the virtual object withinthe CGR environment, etc. For example, a virtual object may berepresented as a 2D object (e.g., an application icon, an applicationwindow, an image, a user interface of an application, etc.). In otherexamples, the virtual object may be represented as a 3D object withinthe CGR environment. In some embodiments, a first representation of avirtual object may be a 3D object including particular content, and asecond, different representation of the virtual object may be a 3Dobject including different content from the particular content in thefirst representation.

In some embodiments, a representation of a virtual object within the CGRenvironment may include audio characteristics. For example, onerepresentation may include particular sounds, noises, spoken words,etc., and a second representation may include different sounds, noises,spoken words, etc. In some cases, the representation of a virtual objectmay also include the level of sound, in which one representation of avirtual object may include one level of sound, and a differentrepresentation may include a higher or lower level of sound.

In accordance with the above, when virtual object 210 is located, atleast partially on location 220, whether by being moved or dragged tolocation 220 or by being displayed on location 220, virtual object 210is displayed as a 2D window on display 240, (e.g., by electronic device200) based on a determination that location 220 is associated with a usecontext that is satisfied by a determination that location 220 is ondisplay 240, display 240 being an electronic device.

In some embodiments, virtual object 210 may be configured such that user202 may interact with virtual object 210. Interaction with virtualobject 220 may be via input sensors, as described above, configured todetect a user input to interact with virtual objects of CGR environment290. In some embodiments, the input sensors may include a mouse, astylus, touch-sensitive surfaces, image-sensors (e.g., to performhand-tracking), etc., which may be configured to allow user 202 to grab,move, drag, click, select and/or otherwise select virtual object 210. Assuch, in embodiments, a request to move virtual object 210 to a locationwithin CGR environment 290 may be received.

In the example shown in FIG. 2A, a request to move virtual object 210from location 220 to another location within CGR environment 290 mayinclude user 202 grabbing or otherwise selecting virtual object 210 formoving from location 220, and may cause virtual object 210 to departlocation 220. In some embodiments, as soon as virtual object 210 isremoved from a location (e.g., location 220), the current representationof virtual object 210 may change. For example, as soon as virtual object210 is removed from location 220, the current representation of virtualobject 210 as a 2D window of a multimedia presentation application maybe changed to another representation. In some implementations, thecurrent representation of virtual object 210 may be changed to sometransitional representation, which may not be associated with aparticular use context, but rather may be a default representationindicating that virtual object 210 is transitioning from one location toanother. In other implementations, the current representation of virtualobject 210 may not be changed when virtual object 210 is removed from alocation but, instead, the current representation of virtual object 210may remain unchanged until the virtual object is positioned in anotherlocation which is determined to be associated with a use context forwhich a different representation of virtual object 210 may be determinedto be displayed. In this case, the current representation of virtualobject 210 may be maintained during transit of virtual object 210 fromthe current location to the new location.

FIG. 2B shows an example of virtual object 210 displayed (e.g., byelectronic device 200) on location 224. In this example, in response tothe request to move the virtual object to location 224, at least one usecontext corresponding to location 224 may be determined. For example,location 224 may correspond to a location on desktop 242. In this case,it may be determined that location 224 is associated with a use contextthat is satisfied by the type of location of location 224 (e.g., thetype of surface, the air), location 224 being a location on desktop 242(e.g., a location on a horizontal plane). In alternative or additionalembodiments, location 224 on desktop 242 may be determined to be alocation in which virtual object 210 may be used, e.g., by user 202, tomake notes regarding a multimedia presentation. In either case, whetherbecause location 224 is a location on a desktop or because location 224is a location in which the virtual object may be used to makeannotations to a multimedia presentation, virtual object 210 may berepresented as a 3D object (e.g., a notepad, notebook, book, or anyother 3D representation) configured to facilitate a user annotatingand/or making notes on the multimedia presentation.

Although not illustrated, virtual object 210 may be moved from location224 on desktop 242 to location 228 also on desktop 242. In embodiments,the representation (e.g., the 3D virtual notepad) of virtual object 210may remain the same on location 228 as in location 224, as bothlocations may be associated with the same use context. Alternatively,although both locations 224 and 228 are on desktop 242 (e.g., the sametype of surface), the representation of the virtual object when onlocation 228 may be different than the representation when on location224. For example, the representation of virtual object when on location228 may be of a different size (e.g., smaller or larger) or may bedifferently oriented, than the representation when on location 224because location 228 may be determined to not be able to accommodate thesize and/or orientation of the representation of virtual object 210 whenon location 224. In some embodiments, different locations within thesame type of surface (e.g., different locations on desktop 242, on wall222, etc.) may be configured for different use contexts. For example, aparticular location on desktop 242 may be configured with a use contextin which the representation of virtual object 210 may be on a particularlanguage, and another location on desktop 242 may be configured with ause context in which the representation of virtual object 210 may be ona different language.

FIG. 2C shows an example of virtual object 210 displayed on location222. For example, a request to move virtual object 210 to location 222may be received. The request may include a request to move virtualobject 210 from any other location within CGR environment 290 (e.g.,location 220, location 224, location 226, etc.). In response to therequest, virtual object 210 may be moved to location 222, and arepresentation of virtual object 210 to be displayed at location 222 maybe determined. In this example, in response to the request to move thevirtual object to location 222, at least one use context correspondingto location 222 may be determined. For example, location 222 maycorrespond to a location on a vertical plane (e.g., a wall) of CGRenvironment 290. In this case, it may be determined that location 222 isassociated with a use context that is satisfied by the type of locationof location 222 (e.g., the type of surface), location 222 being alocation on a wall of CGR environment 290. In alternative or additionalembodiments, location 222 on a wall of CGR environment 290 may bedetermined to be a location in which virtual object 210 may be used topresent a multimedia presentation. In either case, whether becauselocation 222 is a location on a wall or because location 222 is alocation in which the virtual object may be used to present themultimedia presentation, virtual object 210 may be represented (e.g.,displayed by electronic device 200) as a large window object configuredto facilitate presenting the multimedia presentation. For example, thelarge window object may be a 2D window, or a 3D representation of alarge monitor, displayed as fixed against the wall. In some embodiments,the size of the large window object against the wall may be determinedbased on the distance of the wall against which the large window objectis displayed relative to the location of user 202 within CGR environment290.

In some embodiments, the content (e.g., the information and/orarrangement of information) of the representation of virtual object 210on location 222 may be different than the content in the representationsof virtual object 210 at other locations. For example, while at location224, the 3D notepad used as the representation of virtual object 210 mayinclude information arranged in a specific arrangement within the 3Dnotepad. While at location 222, the large window display against thewall used as the representation of virtual object 210 may includedifferent information, which may be arranged in a different arrangement,within the large window display.

FIG. 2D shows an example of virtual object 210 displayed (e.g., byelectronic device 200) on location 226. For example, a request to movevirtual object 210 to location 226 may be received. The request mayinclude a request to move virtual object 210 from any other locationwithin CGR environment 290 (e.g., location 220, location 222, location224, location 228, etc.). In response to the request, virtual object 210may be moved to location 226 and a representation of virtual object 210to be displayed (e.g., by electronic device 200) at location 226 may bedetermined. In this example, in response to the request to move virtualobject to location 226, at least one use context corresponding tolocation 226 may be determined. For example, location 226 may correspondto a location on a horizontal plane (e.g., the floor) of CGR environment290. It is noted that, in this example, location 226 corresponds to alocation on a horizontal plane that is of a different type than thehorizontal plane corresponding to location 224, which is a location ondesktop 242. In this case, it may be determined that location 226 isassociated with a use context that is satisfied by the type of locationof location 226 (e.g., the type of surface), location 226 being alocation on the floor of CGR environment 290. In alternative oradditional embodiments, location 226 on the floor of CGR environment 290may be determined to be a location in which virtual object 210 may beused to at least partially immersively (e.g., from a first-person-viewmode) present a multimedia presentation. In either case, whether becauselocation 226 is a location on a wall or because location 226 is alocation in which the virtual object may be used to at least partiallyimmersively present a multimedia presentation, virtual object 210 may berepresented as a 3D podium placed on, or near, location 226, the podiumconfigured to facilitate user 202 presenting the multimedia presentationfrom the podium. In some embodiments, the representation of virtualobjection 210 at location 226 may include content 212 related to themultimedia presentation (e.g., notes, annotations, presentation content,etc.), and may be presented on top of the podium where user 202 mayperceive content 212.

FIG. 2E shows an example of virtual object 210 being displayed (e.g., byelectronic device 200) on fully-immersive mode. In some embodiments, aparticular location may be associated with a fully-immersive usecontext. For example, a location, such as location 226 on the floor ofCGR environment 290, may be associated with a use context in which thepresentation is to be presented as a fully immersive experience. Inresponse to the request to move virtual object 210 to location 226,virtual object may be moved to location 226, and a fully-immersiverepresentation of virtual object 210 may be displayed. In this case,displaying virtual object 210 as a fully-immersive representation mayinclude displaying the entire CGR environment 290 as a virtualauditorium configured to present the multimedia application.

In some embodiments, a representation of virtual object 210 associatedwith a particular use context may be displayed without having to movethe virtual object to a particular location. For example, with referenceback to FIG. 2A, in some embodiments, an affordance 214 may be presentedwithin CGR environment 290. Affordance 214 may be a virtual object(e.g., a button, an affordance, a user-interface element, an interactiveelement, etc.) configured to allow interaction by a user (e.g., user202). Affordance 214 may correspond to at least one use context. In someembodiments, affordance 214 may also be associated with virtual object214 (e.g., associated with the particular application of virtual object214 such as multimedia presentation, calculator, weather, etc.). Whenuser 202 selects affordance 214 for virtual object 210, the use contextcorresponding to affordance 214 may be considered to be satisfied andmay cause the associated representation (e.g., the representation ofvirtual object 210 associated with the use context) to be displayed. Forexample, where affordance 214 corresponds to the use context associatedwith location 224 (e.g., desktop), as shown in FIG. 2B, a representationof virtual object 210, as a 3D notepad, may be displayed by electronicdevice 200. In some cases, the representation of virtual object 210 maybe displayed at the location associated with the use context (e.g.,without having to move virtual object 210 from its current location tothe location corresponding with the use context associated withaffordance 214), or may be displayed at whichever location virtualobject 210 is currently being displayed. In some embodiments, displayingthe representation of virtual object 210 at the location associated withthe use context corresponding to affordance 214 may include movingvirtual object 210 from its current location to the location associatedwith the use context. In these cases, the moving of virtual object 210to the location associated with the use context may be animated.

In another example, where affordance 214 corresponds to the use contextassociated with a fully-immersive use context, as shown in FIG. 2E, arepresentation of virtual object 210 as a fully-immersive experience maybe displayed by electronic device 200 in response to user 202 selectingaffordance 214.

In some embodiments, affordance 214 may include a plurality ofaffordances, each affordance in the plurality of affordancescorresponding to a particular use context. In these embodiments, eachaffordance in the plurality of affordances may be a selectableaffordance that, when selected, may cause the corresponding use contextto be considered satisfied and may cause the associated representation(e.g., the representation of virtual object 210 associated with thesatisfied use context) to be displayed in accordance with the foregoing.

It is noted that although the present disclosure describes embodimentsin which a virtual object is displayed on a single location within theCGR environment at a time, this is done for illustrative purposes andshould not be construed as limiting in any way. Indeed, in someembodiments, separate and, in some cases, different representations ofthe same virtual object may be displayed at more than one locationwithin the CGR environment concurrently. In embodiments, the separaterepresentations at the different locations may all be different (e.g.,may include different information or may have different shapes and/orforms as described above), or some of the representations at thedifferent locations may be the same while other representations at otherlocations may be different. In some embodiments, a change to theconfiguration of the virtual object (e.g., a change to an applicationassociated with the virtual object) may trigger a change to all therepresentations at all the locations or may trigger a change to somerepresentations at some locations but not all representations at alllocations. In some cases, a change to a representation at one locationwithin the CGR environment (e.g., a change caused in response to userinteraction and/or caused by a change in the associated application) maytrigger at least one change to at least one representation of thevirtual object at another location(s) within the CGR environment.

FIGS. 3A-3C illustrate an example of functionality for controlling arepresentation of a virtual object based on a use context associatedwith a location within the CGR environment in accordance with aspects ofthe present disclosure. In particular, FIG. 3A illustrates user 202wearing electronic device 200, which may be configured to allow user 202to view CGR environment 290. In some embodiments, electronic device 200may be similar to electronic device 100 a described above with referenceto FIGS. 1A and 1B.

CGR environment 290 includes display 340, which may be a physicaldisplay or a virtual representation of a display. In any case, arepresentation of virtual object 310 may be displayed on location 320(e.g., by electronic device 200), which is a location on display 340. Inthe example illustrated in FIG. 3A, virtual object 310 may be acalculator application. In this case, location 320 may be determined tocorrespond to at least one use context (e.g., a type of location,surface, material, etc., and/or a type of use of the virtual object atthe location). For example, location 320 may be determined to be alocation on an electronic device (e.g., a physical device or acomputer-generated simulation of a physical device) of CGR environment290. In this case, it may be determined that location 320 is associatedwith a use context that is satisfied by the type of location of location320 (e.g., the type of surface), location 320 being a location on anelectronic device. Based on the determination that location 320 is alocation on an electronic device, virtual object 310 may be displayed asa 2D window or widget of the calculator application on display 340(e.g., by electronic device 200). Thus, as will be appreciated, therepresentation of virtual object 310 at location 320 is based on the usecontext corresponding to location 320.

FIG. 3B shows user 202 interacting with virtual object 310 at location320. The interaction of user 202 with virtual object 310 at location 320may include a request to move virtual object 310 to another location(e.g., location 324). FIG. 3C shows virtual object 310 having been movedto location 324 in response to the request to move virtual object 310.In this example, at least one use context associated with location 324may be determined. For example, location 324 is a location on desktop342. In this case, it may be determined that location 324 is associatedwith a use context that is satisfied by the type of location of location324 (e.g., the type of surface), location 324 being a location ondesktop 342 (e.g., a location on a horizontal plane). In alternative oradditional embodiments, location 324 on desktop 342 may be determined tobe a location in which virtual object 310 (e.g., a calculatorapplication) may be used, e.g., by user 202, to manipulate thecalculator application in such a manner as to make entries into thecalculator application as in a real-world physical calculator, forexample by using user 202 hands or virtual representations thereof. Ineither case, whether because location 324 is a location on a desktop orbecause location 324 is a location in which the virtual object may beused to make entries into the calculator using a user's hand or virtualrepresentations thereof, virtual object 310 may be represented as a 3Dobject (e.g., a 3D representation of a physical calculator) configuredto facilitate a user making entries into the calculator application.

FIGS. 4A-4C illustrate another example of a representation of a virtualobject of a CGR environment based on a use context associated with alocation of the virtual object within the CGR environment in accordancewith aspects of the present disclosure. In particular, FIG. 4Aillustrates user 202 wearing electronic device 200, which is configuredto allow user 202 to view CGR environment 290. As mentioned above, insome embodiments, electronic device 200 may be similar to electronicdevice 100 a described above with reference to FIGS. 1A and 1B.

CGR environment 290 includes display 440. As described above, display440 may be a physical display or a virtual representation of a display.A representation of virtual object 410 may be displayed by electronicdevice 200 on location 420, which is a location on display 440. In theexample illustrated in FIG. 4A, virtual object 410 may be an applicationfor presenting an interactive and/or animated robot. It will beappreciated that the description of an animated robot herein is forillustrative purposes only and should not be construed as limiting inany way. Indeed, the techniques herein are applicable to any applicationthat may be represented as a virtual object within a CGR environment. Inthis example, location 420 may be determined to be a location on arepresentation of an electronic device (e.g., a representation of adisplay of a physical computer). Based on the determination thatlocation 420 is a location on a representation of an electronic device,virtual object 410 may be displayed (e.g., by electronic device 200) asa 2D window or widget on display 440.

FIG. 4B shows virtual object 410 having been moved to location 424. Inaspects, virtual object 410 may be moved to location 424 in response toa request by a user (e.g., a user interacting with virtual object 410 todrag or otherwise cause to move virtual object 410 to location 424. Inthis example, at least one use context associated with location 424 maybe determined. For example, location 424 is a location on desktop 442.In this case, it may be determined that location 424 is associated witha use context that is satisfied by the type of location of location 424(e.g., the type of surface), location 424 being a location on desktop442 (e.g., a location on a horizontal plane). Based on the use contextcorresponding to location 424, virtual object 410 may be represented(e.g., displayed by electronic device 200) as a 3D object (e.g., a 3Drepresentation of an animated robot). In embodiments, the representationof virtual object 410 when at location 424 may include differentfunctionality than the representation of the virtual object when atlocation 420. For example, the animated 3D robot on desktop 442 may beconfigured to move around desktop 442 in more than one axis. In additionor in the alternative, the animated 3D robot on desktop 442 may be ableto rotate about its own axis. Additionally, or alternatively, theanimated 3D robot on desktop 442 may be configured to be of a largersize than when in location 420.

FIG. 4C shows virtual object 410 having been moved to location 426. Inaspects, virtual object 410 may be moved to location 426 in response toa request by a user (e.g., a user interacting with virtual object 410 todrag or otherwise cause to move virtual object 410 to location 426. Inthis example, at least one use context associated with location 426 maybe determined. For example, location 426 is a location on the floor ofCGR environment 290. It is noted that, in this example, location 426corresponds to a location on a horizontal plane that is of a differenttype than the horizontal plane corresponding to location 424, which is alocation on desktop 442. In this case, it may be determined thatlocation 426 is associated with a use context that is satisfied by thetype of location of location 426 (e.g., the type of surface), location426 being a location on the floor of CGR environment 290. Based on theuse context corresponding to location 424, virtual object 410 may berepresented (e.g., displayed by electronic device 200) as a 3D object(e.g., a 3D representation of an animated robot) on the floor of CGRenvironment 290. In embodiments, the representation of the virtualobject when at location 426 may be different than when in location 424.For example, the animated 3D robot on the floor of CGR environment 290may be larger than the animated 3D robot at location 424 on desktop 442.In addition, the animated 3D robot on the floor of CGR environment 290may be configured to move at a faster rate than the animated 3D robot atlocation 424 on desktop 442.

In some embodiments, some locations within CGR environment 290 may notbe associated with a use context for particular applications or may beprohibited locations with respect to a virtual object associated with aparticular application. For example, location 422 may be a location on avertical plane (e.g., a wall) of CGR environment 290. In this example,location 422 may not have an associated use context. If user 202attempts to move virtual object 210 to location 422, the move may not beallowed, as, e.g., a 3D robot may not be able to navigate on a verticalsurface. Alternatively, a default representation of the virtual objectmay be displayed (e.g., a 2D image or a 2D application window.

FIG. 5 is a flow diagram illustrating method 500 for controlling arepresentation of a virtual object of a CGR environment based on a usecontext associated with a location of the virtual object within the CGRenvironment. In some embodiments, method 500 may be performed by system100 or a portion of system 100. In some embodiments, method 500 may beperformed by one or more external systems and/or devices. In someembodiments, method 500 may be performed by system 100 (or a portion ofsystem 100) in conjunction with one or more external systems and/ordevices.

At block 502, the system displays, via a display of an electronic device(e.g., a wearable electronic device, an HMD device, etc.), a firstrepresentation of a virtual object at a first location within a CGRenvironment. For example, a first representation of a virtual object maybe displayed via a first display (e.g., a left eye display panel) orsecond display (e.g., a second eye display panel) of an electronicdevice.

In embodiments, the first location may correspond to a first use contextof a plurality of use contexts. In embodiments, the plurality of usecontexts may include a use context related to a type of surface (e.g., adesk, a wall, a computer screen, a floor, etc.) and/or the type ofmaterial (e.g., sand, grass, concrete, carpet, etc.), that the virtualobject will be placed on, and/or a use context that corresponds to howthe virtual object will be used (e.g., manipulated, interacted with) ordisplayed (e.g., presented) in the first location of the CGRenvironment. In some embodiments, the system may be a part of theelectronic device, or the electronic device may be a portion of thesystem.

In some embodiments, when the representation of the virtual object isdisplayed at the first location, the representation of the virtualobject may be displayed on a first type of surface (e.g., a desk, awall, a computer screen, a floor, etc.) and the representation of thevirtual object may be displayed based on the first location (e.g., thetype of surface that corresponds to the first location). In someembodiments, one or more of the plurality of use contexts may bepredefined. For example, one or more of the plurality of use contextsmay be predefined based on a particular application corresponding to thevirtual object. In some embodiments, a first application may have afirst number of predefined use contexts, and a second application mayhave a second number of predefined use contexts that is different fromthe first number of predefined use contexts. In some embodiments, thesecond application may have a different use context than the firstapplication, or vice-versa.

At block 504, the system receives a request to move the firstrepresentation, within the CGR environment, to a second location that isdifferent from the first location. In some embodiments, the request maybe received or detected by the system, based on detecting movement ofthe first representation from the first location to the second location.In some embodiments, one or more user inputs may be detected and, inresponse to detecting these user inputs, the system may receive therequest to move the representation to the second location. In someembodiments, the request to move the first representation from the firstlocation to a second location may be received based on one or moredeterminations by an outside application, where based on the one or moredeterminations, the request to move the first representation from thefirst location to the second location is received.

At block 506, in response to receiving the request and in accordancewith a determination that the second location corresponds to a seconduse context (e.g., the second use context being different from the firstuse context) of the plurality of use contexts, the system displays, viathe display of the electronic device, at the second location, near thesecond location, and/or on a surface corresponding to the secondlocation, a second representation of the virtual object based on thesecond use context, and/or based on one or more applications associatedwith the virtual object. In embodiments, the second representation maybe different from the first representation. For example, the secondrepresentation may have a different size, shape, user interface objects,functionality, audio characteristics, surface materials, etc., and/ormay be configured with one or more different and/or additionaloperations than the first representation.

In some embodiments, the second use context of the plurality of usecontext may include a use context that is satisfied when a determinationis made that the second location corresponds to a location (e.g., adisplay, screen, a surface or case of an electronic device) on anelectronic device (e.g., a computer, laptop, tablet, phone, display,projector display). In some embodiments, in accordance with thedetermination that the second location corresponds to the second usecontext of the plurality of use contexts, as a part of displaying thesecond representation of the virtual object based on the second usecontext, the system displays, within the CGR environment, a 2Drepresentation of the virtual object on the electronic device. In someembodiments, the second representation of the virtual object may be the2D representation on the electronic device. In some embodiments, thesecond representation may be moved (e.g., dragged off the display of theelectronic device) to a location in the virtual environment thatcorresponds to a physical surface in a physical environment. In someembodiments, the 2D application may be manipulated as being a 3Dapplication on the electronic device. In some embodiments, the seconduse context of the plurality of use context may include a use contextthat is satisfied when a determination is made that the second locationcorresponds to a location on an electronic device (e.g., a computer,laptop, tablet, phone, display, projector display). In theseembodiments, in accordance with a determination that the second locationcorresponds to the second use context of the plurality of use contexts,displaying the second representation of the virtual object based on thesecond use context may include displaying, within the CGR environment, a3D representation on the electronic device. In some embodiments, therepresentation may change depending on the type (e.g., display (e.g.,monitor), tablet, personal computer, laptop) of the electronic device.

In some embodiments, the second use context of the plurality of usecontext may include a use context that is satisfied when a determinationis made that the second location corresponds to a location on a verticalplane (e.g., a wall, a surface that corresponds to a wall-likestructure, a side of a building, a bedroom wall, a fence, etc.). In someembodiments, in accordance with the determination that the secondlocation corresponds to the second use context of the plurality of usecontexts, as a part of displaying the second representation of thevirtual object based on the second use context, the system displays a 2Drepresentation on the vertical plane (e.g., on the wall) within the CGRenvironment. In some embodiments, the second representation of thevirtual object may be the 2D representation on the electronic device. Insome embodiments, the 2D representation displayed on the vertical plane(e.g., on the wall) within the CGR environment may be bigger, may havemore visual content, may include one or more additional (or different)user interface objects than a 2D representation displayed on theelectronic device. In some embodiments, the representation may changedepending on the type (e.g., side of building, bedroom wall, fence) ofvertical plane and/or one or more characteristics of a vertical plane(e.g., virtual or physical), such as size, shape (e.g., circle,rectangular), material (e.g., brick, wood, metal), texture (e.g., rough,abrasive), color, opacity, etc.

In some embodiments, the size of the second representation may be basedon a distance between the display of the electronic device and thevertical plane within the CGR environment. In some embodiments, the 2Drepresentation may be smaller when the vertical plane is closer to thedisplay of the electronic device and larger when the vertical plan isfarther away from the display of the electronic device. In someembodiments, the size of the 2D representation may be maintained as theuser moves farther away or closer to the 2D representation after the 2Drepresentation is initially displayed. In some embodiments, the size ofthe 2D representation may be changed as the user moves farther away orcloser to the 2D representation after the 2D representation is initiallydisplayed. In some embodiments, the size of the 2D representation may bebased on whether the distance is in a certain category (e.g., categoriesof distance (e.g., far away, close, average distance), where eachcategory of distances corresponds to a different size representation(e.g., extra-large, small, medium)).

In some embodiments, the second use context of the plurality of usecontext includes a use context that is satisfied when a determination ismade that the second location corresponds to a location on a horizontalplane (e.g., a desktop, table, countertop, shelf, floor, an elevatedhorizontal plane, a horizontal plane that is above another horizontalplane, a horizontal plane that is not elevated, etc.) within the CGRenvironment. In some embodiments, in accordance with the determinationthat the second location corresponds to the second use context of theplurality of use contexts, as a part of displaying the secondrepresentation of the virtual object based on the second use context,the system may display a 3D representation on the horizontal planewithin the CGR environment. In some embodiments, the secondrepresentation of the virtual object may be the 3D representation on thehorizontal plane. In some embodiments, the representation may changedepending on the type (e.g., a desktop, table, countertop, shelf) of ahorizontal plane and/or one or more characteristics of horizontal plane(e.g., virtual or physical), such as size, shape (e.g., circle,rectangular), material (e.g., brick, wood, metal), texture (e.g., rough,abrasive), color, opacity, etc.

In some embodiments, in accordance with a determination that thehorizontal plane is a horizontal plane of a first type, the 3Drepresentation may be a representation of a first size. In someembodiments, in accordance with a determination that the horizontalplane is a horizontal plane of a second type, the 3D representation maybe a representation of a second size that is different from (e.g.,greater than) the first size. In embodiments, the first and second typeof horizontal planes may be selected from types of horizontal planesthat may include, for example, a predominantly horizontal plane, astructure that is a horizontal plane, a floor, a side walk, grass, lawn,a surface that one or more people are standing, a non-elevatedhorizontal plane, a horizontal plane that is below another horizontalplane within the CGR environment, etc.

In some embodiments, the 3D representation displayed on the horizontalplane of the first type (e.g., desktop, table, countertop, shelf) withinthe CGR environment may be bigger, may have more visual content, mayinclude one or more additional (or different) user interface objectsthan a 3D representation displayed on the horizontal plane of the secondtype (e.g., floor, sidewalk, grass, lawn, a surface that one or morepeople are standing on).

In some embodiments, the second use context of the plurality of usecontexts may include a use context that is satisfied when maximized viewcriteria are satisfied. For example, maximized view criteria may besatisfied when a user interface element (e.g., button, affordance,and/or any other interactive element) is selected, based on a room wherethe application may be running, based on the second location (e.g., alocation where the virtual object is moved to or dropped), a place on abody part (e.g., place on hand) of a user of the device that correspondsto the maximized criteria being satisfied, a gesture, etc. In theseembodiments, as a part of displaying the second representation of thevirtual object based on the second use context, the system displays aplurality of representations of virtual objects on a plurality of planeswithin the CGR environment. In some embodiments, displaying a pluralityof representations of virtual objects on a plurality of planes withinthe CGR environment may include changing one or more aspects of thephysical environment and/or CGR environment to create a fully orpartially immersive experience. For example, a room (e.g., physical orvirtual) within the CGR environment may be turned into a virtualauditorium when the application is a presentation application, may beturned into a virtual sports venue (e.g., football stadium) when theapplication is a sports viewing application (e.g., fantasy sportsapplication, live sports application), may be turned into a virtualstore when shopping on a shopping application, etc. In some embodiments,the maximized view may be displayed via a companion application (e.g.,fantasy sports application, live sports application, shoppingapplication, presentation application, etc.). In some embodiments, thecompanion application may correspond to the virtual object and/or may bea companion application to an application that corresponds to thevirtual object.

In some embodiments, the selectable virtual object that corresponds to amaximized view affordance may be displayed (e.g., a selectable virtualobject that is displayed currently with a representation, such as thefirst representation, of the virtual object). In some embodiments, themaximized view criteria may include a criterion that is satisfied whenthe selectable virtual object corresponding to a maximized viewaffordance is selected (e.g., a tap or swipe on the virtual object).

In some embodiments, the determination may be made that the secondlocation corresponds to the second use context of the plurality of usecontexts. In some embodiments, the first representation may includefirst visual content (e.g., representations of text, buttons,audio/video, user interface elements, etc.). In some embodiments, thesecond representation may not include the first visual content.

In some embodiments, the determination may be made that the secondlocation corresponds to the second use context of the plurality of usecontexts. In some embodiments, the first representation may includethird visual content that is displayed at a third size. In someembodiments, the second representation may include the third visualcontent that is displayed at a fourth size that is different from (e.g.,larger or smaller representations of text, buttons, audio/video, userinterface elements, etc.) the third size.

In some embodiments, the determination may be made that the secondlocation corresponds to the second use context of the plurality of usecontexts. In some embodiments, the first representation may include afirst selectable object (e.g., one or more selectable user interfaceelements). In some embodiments, the second representation may notinclude the first selectable object.

In some embodiments, the determination may be made that the secondlocation corresponds to the second use context of the plurality of usecontexts. In some embodiments, the first representation is a fourthsize. In some embodiments, the second representation is a fifth sizethat is different from (e.g., larger or small) the fourth size.

In some embodiments, as a part of displaying the second representationof the virtual object based on the second use context, the system maytransition display of the first representation to display of the secondrepresentation when the first representation is within a predetermineddistance (e.g., a distance that is near the second location, when thefirst representation reaches the second location) from the secondlocation. In some embodiments, when the first representation is movedfrom the first location, display of the first representation ismaintained until the first representation reaches or is within a certaindistance of the second location.

In some embodiments, in accordance with a determination that the secondlocation corresponds to a fourth use context of the plurality of usecontexts, wherein the fourth use context is satisfied when the secondlocation corresponds to a prohibited location (e.g., location prohibitedby an application in which the virtual object corresponds and/or one ormore other applications and/or systems), the system forgoes to display,within the CGR environment, a representation of the virtual object basedon the fourth use context. In some embodiments, even when the secondlocation corresponds to a location that satisfies a use context (e.g.,second use context) but for the prohibition of displaying a differentuse context, the first representation may continue to remain displayedbecause display of a different representation than the firstrepresentation is prohibited and/or display of a representation thatcorresponds to the use context (e.g., second use context), that would besatisfied but for the prohibition of displaying a differentrepresentation, is prohibited.

In some embodiments, in accordance with the determination that thesecond location corresponds to the fourth use context of the pluralityof use contexts, the system may display, within the CGR environment, anindication (e.g., a message or symbol that is displayed to note that arepresentation that corresponds to the fourth use context cannot bedisplayed or is prohibited) that the second location is a prohibitedlocation (e.g., location prohibited by an application in which thevirtual object corresponds and/or one or more other applications and/orsystems).

At block 508, in response to receiving the request and in accordancewith a determination that the second location corresponds to a third usecontext (e.g., the third use context is different from the first usecontext and the second use context) of the plurality of use contexts,the system may display, via the display of the electronic device, at thesecond location (e.g., on a surface corresponding to the secondlocation), a third representation of the virtual object based on thethird use context (and/or based on one or more applications associatedwith the virtual object), where the third representation is differentfrom the first representation and the second representation.

Aspects of the present disclosure are directed to systems and techniquesthat provide functionality for controlling concurrent display ofrepresentations of a virtual object within a CGR environment. Inembodiments, controlling the concurrent display of representations of avirtual object may include displaying a first representation on firstsurface (e.g., physical or virtual surface) of the CGR environment, anddisplaying a second representation on second surface of the CGRenvironment different from the first surface. In embodiments, controlsmay be provided for requesting a display of the second representation ofthe virtual object concurrently with the first representation of thevirtual object.

FIGS. 6A-6C illustrate exemplary techniques for controlling concurrentdisplay of representations of a virtual object within a CGR environmentin accordance with aspects of the present disclosure. In particular,FIG. 6A illustrates user 202 wearing electronic device 200, which isconfigured to allow user 202 to view CGR environment 290. As mentionedabove, in some embodiments, electronic device 200 may be similar toelectronic device 100 a described above with reference to FIGS. 1A and1B.

As illustrated in FIG. 6A, CGR environment 290 includes display 640. Asdescribed above, display 640 may be a physical display or a virtualrepresentation of a display. A first representation 620 of virtualobject 610 may be displayed by electronic device 200 at a first surfaceof the CGR environment. For example, first representation 620 of virtualobject 610 may be displayed on display 640. In the example illustratedin FIG. 6A, first representation 620 is a 2D representation displayed ondisplay 640. In embodiments, first representation 620 may be displayedon any surface (e.g., physical or virtual) within CGR environment 290.First representation 620 may include various graphical elementsassociated with the virtual object. For example, as illustrated, virtualobject 610 is associated with a calculator application and includesvarious graphical elements associated with a calculator application. Itwill be appreciated that exemplifying virtual object 610 using acalculator application is done for illustrative purposes, and it is notintended to be limiting in any way. Therefore, virtual object 610 may beassociated with any other type of application (e.g., calendar,multimedia application, presentation, etc.).

In some embodiments, a control may be provided for requesting a displayof a second representation of virtual object 610. A user (e.g., user202) may request the concurrent display, and the request may be receivedby device 200. The request to display of a second representation ofvirtual object 610 may include a request to display the secondrepresentation of virtual object 610 concurrently with firstrepresentation 620. The control for requesting concurrent display mayinclude any technique for providing a selection (e.g., by user 202). Forexample, in some embodiments, the control for requesting concurrentdisplay may include affordance 611 presented within CGR environment 290.In some embodiments, affordance 611 may be provided within firstrepresentation 620 or may be provided outside first representation 620.In some embodiments, affordance 611 may be a virtual object (e.g., abutton, an affordance, a user-interface element, an interactive element,etc.) displayed within CGR environment 290 and configured to allowinteraction by a user (e.g., user 202). In other embodiments, affordance611 may be a graphical element displayed on a physical display (e.g.,rather than a virtual element).

In embodiments, the control for requesting concurrent display mayinclude a gesture that may include moving or dragging virtual object 610out of display 640. For example, user 202 may perform a gesture (e.g.,using an appendage, an input sensor, etc.) in which virtual object 610may be dragged or moved out of display 640. This dragging gesture may bedetermined to be a request to display the second representation ofvirtual object 610 concurrently with first representation 620. In someembodiments, user 202 may drag virtual object 610 out of display 640 andmay continue dragging virtual object to a location within CGRenvironment 290 where the second representation of virtual object 610 isto be displayed.

In some embodiments, the second representation of virtual object 610 maybe displayed within CGR environment 290 in response to receiving therequest to concurrently display representations of virtual object 610.In embodiments, the request to concurrently display representations ofvirtual object 610 may cause an animation in which the secondrepresentation of virtual object 610 comes out (e.g., pops out) of firstrepresentation 620. This is illustrated in FIG. 6B.

FIG. 6C illustrates second representation 621 of virtual object 610displayed within CGR environment 290 in response to receiving therequest to concurrently display representations of virtual object 610.In embodiments, second representation 621 may be displayed on anysurface (e.g., physical or virtual) within CGR environment 290. Inembodiments, second representation 621 may be separate and/or differentfrom first representation 620. For example, as shown in FIG. 6C, firstrepresentation 620 may be a 2D representation of virtual object 610displayed on display 640, and second representation 621 may be a 3Drepresentation of virtual object 610 displayed outside of display 640,on a second and different surface of CGR environment 290. In someembodiments, a 2D representation of an object (e.g., an object within aparticular application or a particular type of application (e.g., acalculator application or a keynote presentation application, apresentation application, a media or entertainment application, aproductivity application)) may be displayed concurrently with a 3Drepresentation of the object. In some embodiments, the 3D representationmay be displayed with or without a 3D representation of the particularapplication or the particular type of application.

In some embodiments, first representation 620 and second representation621, although associated with the same virtual object, may providedifferent or the same functionalities. For example, first representation620 and second representation 621 may share a common set of UI elements.In this example, first representation 620 may be a 2D representation ofan application (e.g., a calculator) that includes a particular set of UIelements for user interaction with the application. Secondrepresentation 621 may be a 3D representation of an application (e.g., acalculator) that includes the same particular set of UI elements foruser interaction as first representation 620. In some embodiments,however, first representation 620 and second representation 621 may havedifferent sets of UI elements. For example, first representation 620 mayinclude a particular set of UI elements, while second representation 621may include a different set of UI elements. In embodiments, one set ofUI elements in the different sets of UI elements may include at leastone UI element that is not included in the other set of UI elements. Inother embodiments, the different sets of UI elements have no UI elementsin common. As will be appreciated, by providing differentfunctionalities, the concurrent display of representations of a virtualobject provides an improved system, as the system may be configured toadapt a representation of a virtual object with functionality dependenton the type of representation (e.g., a 2D representation or a 3Drepresentation).

In some embodiments, one representation of the virtual object may be avirtual representation, while another representation of the virtualobject may not be a virtual representation. For example, display 640 maybe a physical display, and first representation 620 may be a graphicalrepresentation of virtual object 610 displayed on physical display 640.In this case, first representation 620 may not be a virtualrepresentation in that first representation 620 is actually displayed inthe real-world on the physical display and is perceived by user 202 viathe transparent or translucent display of electronic device 200. In thisexample, second representation 621 may be a virtual representation ofvirtual object 610 in that second representation 621 is not actuallydisplayed in the real-world on a physical display, but it is ratherdisplayed on the display of electronic device 200 and is superimposedover the real-world physical display. In this manner, a user may beprovided with the ability to request display of a 3D representation of avirtual object by interacting with controls provided in a 2Drepresentation of the same virtual object. In some embodiments, firstrepresentation 620 and second representation 621 may both be virtualrepresentations.

In embodiments, modifications to one representation of the virtualobject may selectively cause modifications to another representation ofthe virtual object. For example, while first representation 620 andsecond representation 621 are concurrently displayed, a request tomodify first representation 620 may be received. In embodiments, arequest may be received (e.g., from user 202) to modify firstrepresentation 620, for example, to modify the size, the UI elements,the shape, the theme, etc. In embodiments, the request (e.g., userinput) to modify first representation 620 may cause a correspondingmodification to second representation 621 (e.g., size, UI elements,shape, theme, etc.). In aspects, both first representation 620 andsecond representation 621 may be modified in accordance with the requestto modify. In some embodiments, every time a modification to the firstrepresentation 621 is requested, a corresponding modification is made tosecond representation 621. In other embodiments, a first request tomodify the first representation 621 may cause a correspondingmodification to second representation 621. However, a second request tomodify the first representation 621 may not cause a correspondingmodification to second representation 621. In this case, a modificationto second representation 621 is forgone when receiving the secondrequest to modify first representation 620. It is noted that althoughthe foregoing discussion describes selectively modifying secondrepresentation 621 based on a request to modify first representation620, this is done for illustrative purposes and not by way oflimitation. Thus, the same techniques may be used to selectively modifyfirst representation 620 based on a request to modify secondrepresentation 621.

FIG. 7 is a flow diagram illustrating method 700 for controlling aconcurrent display of representations of a virtual object within a CGRenvironment. In some embodiments, method 700 may be performed by system100 or a portion of system 100. In some embodiments, method 700 may beperformed by one or more external systems and/or devices. In someembodiments, method 700 may be performed by system 100 (or a portion ofsystem 100) in conjunction with one or more external systems and/ordevices.

At block 702, the system displays, via a display of an electronic device(e.g., a wearable electronic device, an HMD device, etc.), a 2Drepresentation of a virtual object at a first surface (and/or location)of a CGR environment. For example, a first representation of a virtualobject may be displayed via a first display (e.g., a left eye displaypanel) or second display (e.g., a second eye display panel) of anelectronic device on a representation of a display within the CGRenvironment. In some embodiments, the first surface may be a virtualsurface within the CGR environment. For example, the first surface maybe a virtual representation of a physical display. In other embodiments,the first surface may be a real-world physical surface of the CGRenvironment. For example, the first surface may be a surface of aphysical display. The 2D representation of the virtual object may be avirtual representation (e.g., a virtual representation superimposed overthe first surface via a translucent display of the electronic device) ormay be a real-world graphical representation (e.g., a real-worldgraphical representation displayed on a real-world physical display).

In some embodiments, the 2D representation of the virtual object mayinclude a set of UI element for user interaction with the virtualobject. In embodiments, the 2D representation of the virtual object mayalso include at least one control for requesting concurrent display of asecond representation of the virtual object.

At block 704, the system receives a request to display a 3Drepresentation of the virtual object concurrently with the 2Drepresentation. In embodiments, the request to concurrently display mayinclude a user input. The request may be input by a user using a controlelement (e.g., a button, an affordance, a user-interface element, aninteractive element, etc.) displayed along with 2D representation (e.g.,within the 2D representation or outside the 2D representation). Forexample, the user may select the control element, and the selection maycause a request for a concurrent display to be received by the system.

In some embodiments, the request to concurrently display the 2Drepresentation and the 3D representation may include a gesture to moveor drag the 2D representation out of, or from, the first surface. Forexample, a user 202 may grab, click, and/or otherwise select (e.g.,using an appendage, an input device, an input sensor, etc.) the 2Drepresentation displayed at the first surface and may move or drag the2D representation away from the first surface. In some aspects, thedragging gesture may be determined to be the request for concurrentdisplay.

In embodiments, the request to display a 3D representation of thevirtual object concurrently with the 2D representation may cause ananimation to be played in which the 3D representation is configured tocome out (or pop out) of the 2D representation. In embodiments, theanimation may include a sound that may be played during the animation.

At block 706, in response to the request for concurrent display, thesystem concurrently displays, via the display of the electronic device,the 2D representation at the first surface and the 3D representation ata second surface of the CGR environment. In embodiments, the secondsurface may be different from the first surface. In embodiments, thesecond surface may be a virtual surface or may be a real-world physicalsurface within the CGR environment. For example, the second surface maybe a physical, real-world surface of a desk, or may be a virtualrepresentation of a surface of a physical desk.

In embodiments, the second surface at which the 3D representation may bedisplayed may be determined by user input. For example, a user may dragthe 2D representation out from the first surface and continue draggingto the second surface. In this manner, the 3D representation may bedisplayed in whichever surface within the CGR environment the dragginggesture stops. In other implementations, for example, where a controlelement in the 2D representation is used to request the concurrentdisplay, the second surface may be predetermined. In someimplementations, the user may, prior to requesting concurrent display,indicate a surface at which the 3D representation is to be displayed.For example, a user may first indicate (e.g., via a user input (e.g.,user input detected using input sensors that may include a mouse, astylus, touch-sensitive surfaces, image-sensors (e.g., to performhand-tracking), etc.)), a surface within the CGR environment, other thanthe first surface. Upon requesting concurrent display, the 3Drepresentation may be displayed at the surface indicated by the user.

In some embodiments, the 3D representation of the virtual object mayinclude a set of UI elements for user interaction. In embodiments, theset of UI elements of the 3D representation may be different than theset of UI elements of the 2D representation. For example, one set of UIelements may include UI elements that are not included in the other setof UI elements.

Various aspects of the present disclosure are directed to systems andtechniques that provide functionality for controlling a representationof a virtual object based on characteristics of an input mechanism. Inembodiments, a representation of a virtual object may be based on acharacteristic of the input mechanism (e.g., movement direction,distance, gesture type, etc. of the input mechanism) with respect to thevirtual object. For example, in embodiments, a representation of avirtual object may be modified or maintained depending on whether aninput mechanism associated with the virtual object is within apredetermined distance from a first representation of the virtualobject. In other embodiments, for example, a representation of a virtualobject may be modified or maintained depending on whether an inputmechanism associated with the virtual object is determined to be movingtowards or away from a first representation of the virtual object. Inyet other embodiments, for example, a representation of a virtual objectmay be modified or maintained depending on whether a gesture associatedwith an input mechanism is determined to indicate a potential forinteraction by a user with a first representation of the virtual object.As will be appreciated, the functionality provided by the systems andtechniques described herein provide for an advantageous system in whichrepresentations of virtual objects may be adapted to characteristics ofinput mechanisms, thereby providing an improved user interface.

FIGS. 8A and 8B illustrate exemplary techniques for controlling arepresentation of a virtual object within a CGR environment based oncharacteristics of an input mechanism in accordance with aspects of thepresent disclosure. In particular, FIG. 8A illustrates CGR environment890, including input mechanism 800 and virtual object 810. Inembodiments, CGR environment 890 may be presented to a user (e.g., user202) wearing an electronic device (e.g., electronic device 200)configured to allow user 202 to view CGR environment 890. As mentionedabove, in some embodiments, electronic device 200 may be similar toelectronic device 100 a described above with reference to FIGS. 1A and1B.

As shown in FIG. 8A, first representation 810 of a virtual object may bedisplayed by electronic device 200. In embodiments, first representation810 may be a 3D representation of the virtual object, and the virtualobject may be associated with a particular application. For example, asillustrated in FIG. 8A, first representation 810 may be associated witha calculator application. It will be appreciated that exemplifying firstrepresentation 810, and other representations of a virtual object, usinga particular application (e.g., a calculator application) is done forillustrative purposes, and it is not intended to be limiting in any way.Therefore, first representation 810 may be associated with any type ofapplication (e.g., calendar, multimedia application, presentation,etc.).

In embodiments, first representation 810 may be configured to facilitatenon-direct interaction between a user and first representation 810. Asused herein, non-direct interaction may refer to a user interaction witha representation of a virtual object that does not directly manipulateelements of the representation of the virtual object. A non-limitingexample of a non-direct interaction may be a user perceiving informationprovided by a user interface (UI) element of the representation of thevirtual object without direct manipulation of the UI element by theuser. In contrast, direct interaction, as used herein, may refer to auser interaction with a representation of a virtual object in which UIelements of the representation of the virtual object representation maybe directly manipulated by the user. For example, the user may push abutton, may interact with an interactive element, may click a selectableitem and/or an affordance, etc.

First representation 810 may include UI elements 811 and 815. Inembodiments, UI element 815 may represent at least one UI elementconfigured to provide (e.g., output) information associated with thevirtual object represented by first representation 810. For example, UIelement 815 may be a display of first representation 810. As such, UIelement 815 may be configured for non-direct interaction such that auser may perceive the output without directly manipulating UI element815. UI element 811 may represent at least one UI element that may beconfigurable to a configuration that facilitates user interaction (e.g.,direct interaction or non-direct interaction). For example, UI element811 may be a button, an affordance, a user-interface element, aninteractive element, etc., and/or any combination thereof. When UIelement 811 is configured to facilitate direct interaction, a user mayselect, click, select, and/or otherwise manipulate UI element 811. Insome embodiments, UI element 811 may be configured to facilitatenon-direct interaction by displaying UI element as a 3D element. In thiscase, the user may perceive UI element 811 as a 3D element.

In embodiments, input mechanism 800 may include a mechanism configuredto facilitate interaction with the representations of the virtualobject. For example, input mechanism may include a mechanism for a user(e.g., user 202) to manipulate at least one element of a representationof the virtual object or to perceive data provided by an element of therepresentation of the virtual object. In embodiments, input mechanism800 may include a representation of an appendage of the user (e.g., afinger, hand, leg, foot, etc.), a user's gaze (e.g., head gaze, eyegaze, etc.), an input device (e.g., a mouse, a stylus, etc.) (e.g., thatis different from the electronic device, that is in operativecommunication with the electronic device, that is physically connectedto (e.g., or a part of) the electronic device), etc. In embodiments, therepresentation of an appendage of the user may include a virtualrepresentation of the appendage and/or may include data representingcharacteristics of the appendage (e.g., location, orientation, distanceto a particular point, etc.) within the CGR environment. In aspects,input mechanism 800 may be detected using input sensors (e.g.,touch-sensitive surfaces, image-sensors, etc.) configured to performhand-tracking, head gaze-tracking, eye gaze-tracking, finger-tracking,etc. As shown in FIG. 8A, input mechanism 800 may include a user'sappendage (e.g., a finger).

As shown in FIG. 8A, and discussed above, first representation 810 maybe displayed within CGR environment 890, and first representation 810may be configured to facilitate non-direct interaction by a user ratherthan direct interaction (e.g., by providing UI elements 811 and 815configured for non-direct interaction). As also shown in FIG. 8A, inputmechanism 800 may be at a current location that is distance 831 fromfirst representation 810. In some embodiments, a predetermined distance830 from first representation 810 may be provided, although in someimplementations, predetermined distance 830 may not be shown within CGRenvironment 890. Predetermined distance 830 may be configured to operateas a threshold, such that when the current location of the inputmechanism is not within predetermined distance 830 from firstrepresentation 810, the displaying of first representation 810 may bemaintained. For example, as distance 831 may be determined to be greaterthan predetermined distance 830, the current location of input mechanism800 may be determined not to be within a predetermined distance 830 fromfirst representation 810.

In embodiments, whether the displaying of first representation 810 maybe modified or maintained may be based on a characteristic of inputmechanism 800. In some embodiments, the characteristic of inputmechanism 800 may include a movement direction, a distance to arepresentation of the virtual object, a gesture type, etc. In accordancewith the determination that the current location of input mechanism 800is not within predetermined distance 830 from first representation 810,the displaying of first representation 810 may be maintained withoutdisplaying another representation of the virtual object. Conversely, aswill be discussed below, and as illustrated in the example shown in FIG.8B, in accordance with a determination that the current location ofinput mechanism 800 is within predetermined distance 830 from firstrepresentation 810, the displaying of first representation 810 may bemodified, and a second representation of the virtual object may bedisplayed. In aspects, the second representation of the virtual objectmay be different from first representation 810.

In some embodiments, the determination of whether the location of inputmechanism 800 is within predetermined distance 830 from firstrepresentation 810 may be performed in response to detecting a movementof input mechanism 800. In these cases, if no movement of inputmechanism 800 is detected, the determination of whether the location ofinput mechanism 800 is within predetermined distance 830 from firstrepresentation 810 may not be performed. In some embodiments, thedetermination of whether the location of input mechanism 800 is withinpredetermined distance 830 from first representation 810 may beperformed when a detected movement is determined to be towards firstrepresentation 810. In these cases, if the movement of input mechanism800 is determined to be away from first representation 810, thedetermination of whether the location of input mechanism 800 is withinpredetermined distance 830 from first representation 810 may not beperformed even though a movement of input mechanism 800 may be detected.

In some implementations, first representation 810 may be initiallydisplayed within CGR environment 890 in response to a determination thatinput mechanism 800 is not within predetermined distance 830 from alocation at which first representation 810 is to be displayed. Forexample, a determination may be made to initially display arepresentation of a virtual object at a first location within CGRenvironment 890. In this example, the first representation of thevirtual object may be configured for non-direct interaction. Further, inthis example, CGR environment 890 may not include any representation ofthe virtual object at the first location, although in some cases atleast one other representation of the virtual object may be displayed atanother location within CGR environment 890. In response to thedetermination to initially display a representation of the virtualobject at the first location within CGR environment 890, a determinationmay be made as to whether the current location of input mechanism 800 iswithin predetermined distance 830 from the first location or not. If itis determined that the current location of input mechanism 800 is notwithin predetermined distance 830 from the first location, the firstrepresentation (e.g., first representation 810) may be displayed at thefirst location. In some embodiments, if it is determined that thecurrent location of input mechanism 800 is within predetermined distance830 from the first location, a second representation (e.g., secondrepresentation 820 described below) configured for direct interactionmay be displayed at the first location.

As shown in FIG. 8B, input mechanism 800 may be moved (e.g., indirection 833) from a previous location (e.g., as illustrated in FIG.8A) to a current location with a distance 832 to first representation810. The movement from the previous location to the current location maybe detected (e.g., using input sensors as described above). In responseto detecting the movement of input mechanism 800 from the previouslocation to the current location, a determination may be made as towhether the current location of input mechanism 800 to firstrepresentation 810 may be within predetermined distance 830 or not. Forexample, distance 832 from the current location of input mechanism 800to the first representation 810 may be compared against predetermineddistance 830. In accordance with a determination that the distance 832is greater than predetermined distance 830, the current location ofinput mechanism 800 may be determined to not be within predetermineddistance 830 from first representation 810. Conversely, in accordancewith a determination that the distance 832 is not greater thanpredetermined distance 830, the current location of input mechanism 800may be determined to be within predetermined distance 830 from firstrepresentation 810.

In embodiments, in accordance with a determination that the currentlocation of input mechanism 800 is within predetermined distance 830from first representation 810, the displaying of first representation810 may be modified. In embodiments, modifying the displaying of firstrepresentation 810 may include ceasing to display first representation810 and displaying second representation 820, where secondrepresentation 820 may be different from first representation 810. Insome embodiments, second representation 820 may be displayed at the samelocation and/or on the same surface where first representation 810 wasdisplayed.

In embodiments, second representation 820 may be configured for directinteraction between the user (e.g., user 202) and second representation820 (e.g., elements of second representation 820). For example, whereasfirst representation 810 includes UI element 811, as shown in FIG. 8A,configured for non-direct interaction (e.g., UI elements displayed asprotruding 3D UI elements), second representation 820 may include UIelement 821 configured for direct interaction. In this example, UIelement 821 may include at least one UI element displayed as flatbuttons, or as 2D elements, where the flat buttons may not protrude fromsecond representation 820. As will be appreciated, a flat 2D UI element(e.g., a 2D button) displayed upon a physical table (e.g., on the sameplane as the physical table, may be more apt to provide physicalfeedback when a user manipulates the 2D element. For example, as theuser manipulates the 2D element, the user receives the feedback providedby the physical table upon which the virtual 2D element I displayed. Inaddition, displaying second representation 820 configured for directinteraction may also encourage the user (e.g., user 202) to interactwith second representation 820.

In some embodiments, modifying first representation 810, which mayinclude displaying second representation 820, may include animating themodification. For example, one of the differences between firstrepresentation 810 and second representation 820 may be that UI element811 of first representation 810 is displayed as protruding 3D UIelements and UI element 821 of second representation 820 is displayed asflat 2D UI elements. In this example, the modification of firstrepresentation 810 may include animating the UI elements such that theprotruding 3D UI elements of first representation 810 are presented asreceding into the flat 2D UI elements of second representation 820. Inembodiments, the animation may also include a sound that may be playedwhile the animation is occurring.

In another embodiment, modifying the first representation of the virtualobject may include moving the first representation to a location closerto the user (e.g., user 202). For example, based on the characteristicof the input mechanism 800 (e.g., the current location of inputmechanism 800 is within a predetermined distance (e.g., predetermineddistance 830) from the current location of the first representation(e.g., first representation 810)), a second representation of thevirtual object may be displayed. In embodiments, the secondrepresentation of the virtual object may be the same as the firstrepresentation but in a location that is closer to the user than thecurrent location of the first representation. In some embodiments, thesecond representation displayed at the new location may be a differentrepresentation of the first representation, for example, in accordancewith the above description.

In further embodiments, the characteristic of the input mechanism onwhich the determination to modify or maintain the first representation810 may be based may include a determination of whether the direction ofthe movement of input mechanism 800 is toward or away from firstrepresentation 810. For example, as shown in FIG. 8B, input mechanism800 may be moved in direction 833, which is a direction toward firstrepresentation 810. In this case, in accordance with the determinationthat the direction of the movement of input mechanism 800 is towardfirst representation 810, the displaying of first representation 810 maybe modified and a second representation (e.g., second representation 820configured to facilitate direct interaction by a user) of the virtualobject may be displayed. Conversely, in accordance with thedetermination that the direction of the movement of input mechanism 800is away from first representation 810, the displaying of firstrepresentation 810 may be maintained without displaying anotherrepresentation (e.g., second representation 820) of the virtual object.In aspects, the second representation of the virtual object may bedifferent than first representation 810.

In yet further embodiments, the characteristic of the input mechanism onwhich the determination to modify or maintain the first representation810 may include a determination of whether a particular type of gesturehas been made by input mechanism 800. In aspects, the particular type ofgesture may be a gesture that may indicate a potential for direct userinteraction. For example, as shown in FIG. 8B, input mechanism 800 maybe a pointing hand. In embodiments, a pointing hand may be considered atype of gesture that indicates a potential for user interaction. As willbe appreciated, a user desiring to interact with a virtual object, suchas a virtual object represented with UI elements for user input, using afinger may do so by forming his or her hand into a pointing hand withthe finger pointing out. In this sense, the pointing hand may indicatethat the user intends or desires to interact with the virtual object. Assuch, when a determination is made that input mechanism has made agesture that indicates a potential for user interaction (e.g., pointinghand, grabbing hand, etc.), a determination may be made to modify acurrent representation configured for non-direct interaction (e.g.,first representation 810) into a representation configured for directinteraction (e.g., second representation 820). In aspects, themodification of the current representation configured for non-directinteraction into a representation configured for direct interaction maybe in accordance with the foregoing description.

In another example, a determination to maintain the displaying of firstrepresentation 810 configured for non-direct interaction may be based ona gesture that does not indicate a potential for user interaction. Forexample, a gesture may be detected that may include the user (e.g., user202) crossing his or her arms, and/or leaning back. In this case, thegesture may be considered a type of gesture that does not indicate apotential for user interaction. As such, when a determination is madethat the user has crossed his or her arms, and/or has leaned back, adetermination may be made to maintain a current representationconfigured for non-direct interaction (e.g., first representation 810)without displaying a representation configured for direct interaction(e.g., second representation 820). In some embodiments, detecting agesture that does not indicate a potential for user interaction maycause a determination to modify a current representation configured fordirect interaction (e.g., second representation 820) into arepresentation configured for non-direct interaction (e.g., firstrepresentation 810).

It is noted that although the foregoing examples, and the examples thatfollow, may be focused on a description of modifications of arepresentation of a virtual object configured for non-direct interactioninto a representation of the virtual object configured for directinteraction, this is done for illustrative purposes and not intended tobe limiting in any way. In some embodiments, a representation of avirtual object configured for direct interaction may be modified into arepresentation of the virtual object configured for non-directinteraction based on characteristics of the input mechanism. Forexample, in some implementations, a display of a representationconfigured for direct interaction (e.g., first representation 810described above) may be modified to display a representation configuredfor non-direct interaction (e.g., second representation 820 describedabove) based on a detected movement of an input mechanism, based on acharacteristic of the input mechanism (e.g., in accordance with adetermination that the location of the input mechanism is not within apredetermined distance from the representation configured for directinteraction (e.g., first representation 810)). As such, the presentdisclosure provides techniques for selectively and dynamicallyconfiguring a representation of a virtual object enhanced interaction(e.g., direct or non-direct) based on the characteristics of the inputmechanism. Thus, the representation of the virtual object may beconfigured for direct or non-direct interaction when it is moreadvantageous based on the characteristics of the input mechanism.

Additionally, although the foregoing discussion describes secondrepresentation 820 as configured for direct interaction with flat 2D UIelements, it will be appreciated that this is done for illustrativepurposes and not by way of limitation. As will be appreciated, arepresentation of a virtual object may be configured for directinteraction by other methods (e.g., orientation, size, angle, shape,color, brightness, language, location, distance, direction, etc.). Forexample, in embodiments, based on a characteristic of the inputmechanism (e.g., in accordance with a determination that the currentlocation of an input mechanism is within a predetermined distance from afirst representation of a virtual object), the displaying of the firstrepresentation may be modified, and the modification may includedisplaying a second representation different from the firstrepresentation. In these embodiments, the second representation mayinclude a different orientation, size, angle, shape, color, brightness,language, location, distance, direction, etc. from the firstrepresentation, where the modification may be configured to allow,encourage, enable, and/or otherwise facilitate direct interaction withthe second representation of the virtual object. Some of theseembodiments will be described in further detail below.

FIGS. 9A and 9B illustrate another example of techniques for controllinga representation of a virtual object within a CGR environment based oncharacteristics of an input mechanism in accordance with aspects of thepresent disclosure. As shown in FIG. 9A, first representation 910 of avirtual object may be displayed via a display of electronic device 200.In embodiments, first representation 910 may be a 3D representation ofthe virtual object, and the virtual object may be associated with aparticular application (e.g., calendar, multimedia application,presentation, etc.), as discussed above. In the example illustrated inFIG. 9A, first representation 910 may be associated with a calculatorapplication.

In embodiments, first representation 910 may be configured to facilitatenon-direct interaction with the associated virtual object. For example,first representation 910 may include UI elements 911 and 915. Inembodiments, UI element 915 may represent at least one UI elementconfigured to provide (e.g., output) information associated with thevirtual object represented by first representation 910. For example, UIelement 915 may be a display (e.g., a virtual display) of firstrepresentation 910. In this case, first representation 910 may beconfigured to facilitate non-direct interaction by a user by beingdisplayed at an orientation that facilitates the user (e.g., user 202)non-direct interaction with UI element 915. For example, firstrepresentation 910 may be displayed at an orientation that includesangle 912. In embodiments, angle 912 may be an angle that is configuredto place first representation 910 at an orientation that enables theuser to see, hear, or otherwise perceive, UI element 915. In thismanner, angle 912 facilitates the user non-direct interaction with UIelement 915. In embodiments, angle 912 may be measured with respect to asurface (e.g., surface 916) on which first representation 910 isdisplayed.

In embodiments, the orientation at which first representation 910 may bedisplayed may be determined based on the location of the user. Forexample, the user's gaze (e.g., head gaze and/or eye gaze) may bedetermined (e.g., by detecting the location of the user's head and/oreyes and then determining the user's gaze), and the determined user'sgaze may then be used to determine an orientation at which to displayfirst representation 910 such that UI elements configured for non-directinteraction (e.g., UI element 915) are facing the user's gaze.

In embodiments, UI element 911 of first representation 910, may beconfigured for non-direct interaction. In this case, UI element 811 maybe displayed as protruding buttons, or as 3D elements, where the flatbuttons may not protrude from first representation 910. In this manner,UI element 911, as shown in FIG. 9A, is not configured for directinteraction.

As shown in FIG. 9A, and as discussed above, first representation 910may be configured to facilitate non-direct interaction by a user ratherthan direct interaction (e.g., by providing protruding 3D UI element 911and by orienting first representation 910 at angle 912). As also shownin FIG. 9A, input mechanism 800 may be at a current location that isdistance 931 from first representation 910. In some embodiments,predetermined distance 930 from first representation 910 may beprovided.

In embodiments, in accordance with a determination that the currentlocation of input mechanism 800 is not within predetermined distance 930from first representation 910, the displaying of first representation910 may be maintained. For example, first representation 910 configuredfor non-direct interaction may continue to be displayed withoutdisplaying another representation of the virtual object and/or withoutmaking changes to first representation 910. Conversely, as will bediscussed below, in accordance with a determination that the currentlocation of input mechanism 800 is within predetermined distance 930from first representation 910, the displaying of first representation910 may be modified and a second representation of the virtual objectmay be displayed. In aspects, the second representation of the virtualobject may be different than first representation 910.

As shown in FIG. 9B, input mechanism 800 may be moved, e.g., indirection 933, from a previous location (e.g., as illustrated in FIG.9A) to a current location with a distance 932 to first representation910. The movement from the previous location to the current location maybe detected (e.g., using input sensors as described above). In responseto detecting the movement of input mechanism 800 from the previouslocation to the current location, a determination may be made as towhether the current location of input mechanism 800 to firstrepresentation 910 may be within predetermined distance 930 or not. Forexample, distance 932 from the current location of input mechanism 800to the first representation 910 may be compared against predetermineddistance 930. In accordance with a determination that the distance 932is greater than predetermined distance 930, the current location ofinput mechanism 800 may be determined to not be within predetermineddistance 930 from first representation 910. Conversely, in accordancewith a determination that the distance 932 is not greater thanpredetermined distance 930, the current location of input mechanism 800may be determined to be within predetermined distance 930 from firstrepresentation 910.

In embodiments, in accordance with a determination that the currentlocation of input mechanism 800 is within predetermined distance 930from first representation 910, the displaying of first representation910 may be modified. In embodiments, modifying the displaying of firstrepresentation 910 may include ceasing to display first representation910 and displaying second representation 920, where secondrepresentation 920 may be different from first representation 910. Insome embodiments, second representation 920 may be displayed at the samelocation and/or on the same surface where first representation 910 wasdisplayed.

In embodiments, second representation 920 may be configured tofacilitate direct interaction by the user (e.g., user 202) with theassociated virtual object. For example, whereas first representation 910is displayed at an orientation with angle 912, which facilitates theuser being able to perceive (e.g., see, hear, etc.) information providedby UI element 915 (e.g., non-direct interaction), second representation920 may be displayed at an orientation that facilitates the userdirectly interacting (e.g., directly manipulating, selecting, clicking,dragging, and/or otherwise selecting) with UI elements of secondrepresentation 920 (e.g., UI element 921). For example, secondrepresentation 920 may be displayed within CGR environment 890 at anorientation that is longitudinal with surface 916. As such, secondrepresentation 920 may be displayed as lying flat on surface 916. Aswill be appreciated, a flat surface may be easier to interact with thanan angled surface. As such, by modifying the representation of thevirtual object from an angled orientation to a flat orientation, orvice-versa, the representation of the virtual object is selectivelyadapted for enhanced direct-interaction based on the characteristics ofthe input mechanism. In some embodiments, second representation 920 maybe displayed at an orientation having a non-zero angle with respect tosurface 916 that is different from angle 912.

In addition, whereas first representation 910 includes UI element 911,as shown in FIG. 9A, configured for non-direct interaction (e.g., UIelements displayed as protruding 3D UI elements, where the protruding 3DUI elements may protrude (or pop out) from first representation 910),second representation 920 may include UI element 921 configured fordirect interaction, as previously described. For example, UI element 921may include at least one UI element displayed as flat 2D UI elementsdisplayed upon a physical object, which facilitates physical feedback asthe user manipulates the 2D UI elements.

In some embodiments, modifying first representation 910, which mayinclude displaying second representation 920, may include animating themodification. For example, the modification of first representation 910may include animating a change in orientation of first representation910 such that first representation 910 is displayed as moving from thecurrent orientation (e.g., angled at angle 912) to the orientation ofsecond representation 920 (e.g., flat on surface 916). In addition, orin the alternative, the modification of first representation 910 mayinclude animating the UI elements such that the protruding 3D UIelements of first representation 910 are presented as receding into theflat 2D UI elements of second representation 920. In embodiments, theanimation may also include a sound that may be played while theanimation is occurring.

FIGS. 10A and 10B illustrate another example of techniques forcontrolling a representation of a virtual object within a CGRenvironment based on characteristics of an input mechanism in accordancewith aspects of the present disclosure. In particular, FIGS. 10A and 10Billustrate an example in which a representation of a virtual object ismodified based on characteristics of an input mechanism, and in whichthe modification includes adding UI elements for user interaction andchanging the size of the representation.

As shown in FIG. 10A, first representation 1010 of a virtual object maybe displayed via a display of electronic device 200. In embodiments,first representation 1010 may be a 3D representation of the virtualobject, and the virtual object may be associated with a particularapplication (e.g., calendar, multimedia application, presentation,etc.), as discussed above. In the example illustrated in FIG. 10A, firstrepresentation 1010 may be associated with a calculator application.

In embodiments, first representation 1010 may be configured tofacilitate non-direct interaction with the associated virtual object.For example, first representation 910 may include UI elements 1012. UIelement 1012 may represent at least one UI element configured to provide(e.g., output) information associated with the virtual objectrepresented by first representation 1010. For example, UI element 1012may be a display (e.g., a virtual display) of first representation 1010.In some embodiments, first representation 1010 may have a size. In someembodiments, first representation 1010 may not include any UI elementsconfigured for user input (e.g., a button, an affordance, auser-interface element, an interactive element, etc.).

As shown in FIG. 10A, and as discussed above, first representation 1010may be displayed within CGR environment 890, and first representation1010 may be configured to facilitate non-direct interaction by a userrather than direct interaction. As also shown in FIG. 10A, inputmechanism 800 may be at a current location that is distance 1031 fromfirst representation 1010. In some embodiments, predetermined distance1030 from first representation 1010 may be provided.

In embodiments, in accordance with a determination that the currentlocation of input mechanism 800 is not within predetermined distance1030 from first representation 1010, the displaying of firstrepresentation 1010 may be maintained. For example, first representation1010 configured for non-direct interaction may continue to be displayedwithout displaying another representation of the virtual object and/orwithout making changes to first representation 1010. Conversely, as willbe discussed below, in accordance with a determination that the currentlocation of input mechanism 800 is within predetermined distance 1030from first representation 1010, the displaying of first representation1010 may be modified and a second representation of the virtual objectmay be displayed. In aspects, the second representation of the virtualobject may be different than first representation 1010.

As shown in FIG. 10B, input mechanism 800 may be moved from a previouslocation (e.g., as illustrated in FIG. 9A) to a current location with adistance 1030 to first representation 1010. The movement from theprevious location to the current location may be detected (e.g., usinginput sensors as described above). In response to detecting the movementof input mechanism 800 from the previous location to the currentlocation, a determination may be made as to whether the current locationof input mechanism 800 to first representation 1010 may be withinpredetermined distance 1030 or not. In accordance with a determinationthat the current location of input mechanism 800 is within predetermineddistance 1030 from first representation 1010, the displaying of firstrepresentation 1010 may be modified. In embodiments, modifying thedisplaying of first representation 1010 may include ceasing to displayfirst representation 1010 and displaying second representation 1020,where second representation 1020 may be different from firstrepresentation 1010. In some embodiments, second representation 1020 maybe displayed at the same location and/or on the same surface where firstrepresentation 1020 was displayed.

In embodiments, second representation 1020 may be configured tofacilitate direct interaction by the user (e.g., user 202) with theassociated virtual object. For example, whereas first representation1010 may not include UI elements 911 configured for user input, secondrepresentation 1020 may include UI element 1021 configured for userinteraction, as previously described. For example, UI element 1021 mayinclude at least one UI element displayed as flat 2D UI elements.

In addition, second representation 1020 may be displayed having a sizethat is different than the size of first representation 1010. Forexample, second representation 1020 may be displayed with a size largerthan the size of first representation 1010. In some embodiments, secondrepresentation 1020 may be displayed with a size smaller than the sizeof first representation 1010.

As previously described, in some embodiments, modifying firstrepresentation 1010, which may include displaying second representation1020 may include animating the modification. For example, themodification of first representation 1010 may include animating a changein size of first representation 1010 such that first representation 1010is displayed as growing or shrinking, as appropriate, from the currentsize to the size of second representation 1020. In addition, or in thealternative, the modification of first representation 1010 may includeanimating the UI elements such that the protruding 3D UI elements offirst representation 910 are presented as receding into the flat 2D UIelements of second representation 920. In embodiments, the animation mayalso include a sound that may be played while the animation isoccurring.

FIGS. 11A and 11B illustrate another example of techniques forcontrolling a representation of a virtual object within a CGRenvironment based on characteristics of an input mechanism in accordancewith aspects of the present disclosure. In particular, FIGS. 11A and 11Billustrate an example in which a representation of a virtual object ismodified based on characteristics of an input mechanism (e.g., a user'sgaze).

As shown in FIG. 11A, first representation 1110 of a virtual object maybe displayed via a display of electronic device 200. In embodiments,first representation 1110 may be a representation of the virtual object,and the virtual object may be associated with a particular application(e.g., calendar, multimedia application, presentation, etc.) asdiscussed above. In the example illustrated in FIG. 11A, firstrepresentation 1110 may be associated with a calendar application.

In embodiments, first representation 1110 may have a size and may bedisplayed at location 1152. In embodiments, first representation 1110may not be configured for user interaction, whether direct or non-directinteraction. For example, the size of first representation 1110 may be asmall size, and the small size may not enable a user to perceive anyinformation from or interact with any UI elements of firstrepresentation 1110. In some embodiments, first representation 1110 maynot include any UI elements.

As shown in FIG. 11A, a gaze 1150 of user 202, wearing electronic device200, may be detected. In aspects, detected gaze 1150 can be a head gaze(e.g., the direction in which the user's head is facing), an eye gaze(e.g., the direction in which the user's eyes are looking), acombination thereof, etc. Gaze 1150 of user 202 may be determined to befocused, placed, or otherwise directed to location 1151, which may bedifferent from location 1152 where first representation 1110 isdisplayed. In aspects, in accordance with the determination that gaze1150 is directed to a location that is different than the location offirst representation 1110, the displaying of first representation 1110,at the current location and having the size, may be maintained withoutdisplaying another representation of the virtual object and/or withoutmaking any changes to first representation 1110.

FIG. 11B shows that gaze 1150 of user 202 has changed to a differentdirection than the direction directed to location 1151. In embodiments,the change in gaze may be detected (e.g., via input sensors). Inresponse to the detected change in the user's gaze, a determination ofthe direction of the new direction of the gaze may be made. For example,it may be determined that the new direction of gaze 1150 may be directedto location 1152. Location 1152 may be the location at which firstrepresentation 1110 is being displayed. In embodiments, in accordancewith a determination that gaze 1150 is directed to a location that isthe same as the location of first representation 1110, the displaying offirst representation 1110 may be modified.

In some embodiments, determining to modify the displaying of firstrepresentation 1110 in accordance with a determination that gaze 1150 isdirected to a location that is the same as the location of firstrepresentation 1110 may include a determination that the gaze 1150 hasremained directed to the location that is the same as the location offirst representation 1110 for at least a predetermined period of time.When it is determined that gaze 1150 has remained directed to thelocation that is the same as the location of first representation 1110for a period of time that is less than the predetermined period of time(e.g., the direction of gaze 1150 is moved to a different directionbefore the predetermined period of time expires), the displaying offirst representation 1110 may not be modified, but instead may bemaintained without displaying another representation of the virtualobject and/or without making any changes to first representation 1110.When it is determined that gaze 1150 has remained directed to thelocation that is the same as the location of first representation 1110for a period of time that is at least the same as the predeterminedperiod of time (e.g., the direction of gaze 1150 does not move to adifferent direction before the predetermined period of time expires),the displaying of first representation 1110 may be modified.

In embodiments, modifying the displaying of first representation 1110may include ceasing to display first representation 1110 and displayingsecond representation 1120, where second representation 1120 may bedifferent from first representation 1110. In some embodiments, secondrepresentation 1120 may be displayed at the same location and/or on thesame surface where first representation 1120 was displayed.

In embodiments, second representation 1120 may be different from firstrepresentation 1110, and second representation 1120 may be configured tofacilitate interaction by the user (e.g., user 202). For example, secondrepresentation 1120 may be configured to include UI elements 1112. UIelements 1112 may include at least one UI element configured for userinteraction, such as a display. In some embodiments, secondrepresentation 1120 may alternatively or additionally have a sizedifferent than the size of first representation 1110. For example,second representation 1120 may have a size that is larger or smallerthan the size of first representation 1110. In embodiments, the size ofsecond representation 1120 may be based on a distance between thelocation of second representation 1120 (e.g., location 1152) and thelocation of the user's head and/or eyes (e.g., location 1153).

In some embodiments, second representation 1120 may be configured fornon-direct interaction, but may not be configured fordirect-interaction. For example, second representation 1120 may notinclude any UI elements configured for direct interaction with a user(e.g., a button, an affordance, a user-interface element, an interactiveelement, etc.). In this case, the techniques described above withrespect to FIGS. 8A, 8B, 9A, 9B, 10A, and 10B may be used to selectivelymodify second representation 1120 into a configuration for directinteraction based on a characteristic of an input mechanism (e.g., arepresentation of an appendage, a mouse, a stylus, etc.) in accordancewith the disclosure herein. In this manner, a representation of avirtual object may be selectively and dynamically modified from anon-interaction configuration to a non-direct interaction configurationbased on a characteristic of an input mechanism (e.g., a user's gaze),and then may be further modified from the non-direct interactionconfiguration to a direct interaction configuration based on anothercharacteristic of the input mechanism or based on a characteristic ofanother input mechanism (e.g., a representation of an appendage, inputdevice, etc.).

FIGS. 12A and 12B illustrate another example in which a representationof a virtual object within a CGR environment is modified based on auser's gaze. In particular, FIG. 12A shows user 202 wearing electronicdevice 200 configured to allow user 202 to view CGR environment 890. Asshown in FIG. 12A, first representation 1210 of a virtual object may bedisplayed via a display of electronic device 200 at location 1251 andwith a particular size. In aspects, location 1251 may be on a wall ofCGR environment 890. In embodiments, first representation 1210 may be arepresentation of the virtual object, and the virtual object may beassociated with a particular application (e.g., calendar, multimediaapplication, presentation, etc.) as discussed above. In the exampleillustrated in FIG. 12A, first representation 1210 may be associatedwith a calendar application. In embodiments, first representation 1210may not be configured for user interaction, whether direct or non-directinteraction. For example, the size of first representation 1210 may be asmall size, and the small size may not enable a user to perceive anyinformation from or interact with any UI elements of firstrepresentation 1210.

As shown in FIG. 12A, a gaze 1250 of user 202 may be determined to bedirected to location 1252, which may be different from location 1251where first representation 1210 is displayed. In aspects, in accordancewith the determination that gaze 1250 is directed to a location that isdifferent than the location of first representation 1210, the displayingof first representation 1210 may be maintained without displayinganother representation of the virtual object and/or without making anychanges to first representation 1210.

FIG. 12B shows that gaze 1250 of user 202 has changed to a differentdirection than the direction directed to location 1252. In embodiments,the change in gaze may be detected (e.g., via input sensors). Inresponse to the detected change in the user's gaze, a determination ofthe direction of the new direction of the gaze may be made. For example,it may be determined that the new direction of gaze 1250 may be directedto location 1251. Location 1251 is the location at which firstrepresentation 1210 is being displayed. In embodiments, in accordancewith a determination that gaze 1250 is directed to a location that isthe same as the location of first representation 1210, the displaying offirst representation 1210 may be modified. For example, firstrepresentation 1210 may cease to be displayed, and second representation1220 may be displayed, where second representation 1220 may be differentfrom first representation 1210. In some embodiments, secondrepresentation 1220 may be displayed at the same location and/or on thesame surface where first representation 1210 was displayed.

In embodiments, second representation 1220 may be configured to includeUI elements 1221. UI elements 1221 may include at least one UI elementconfigured for user interaction, such as a display. In some embodiments,second representation 1220 may alternatively or additionally have a sizedifferent than the size of first representation 1210. For example,second representation 1220 may have a size that is larger or smallerthan the size of first representation 1210. In embodiments, the size ofsecond representation 1220 may be based on a distance between thelocation of second representation 1220 (e.g., location 1251) and thelocation of the user's head and/or eyes. In some embodiments, secondrepresentation 1120 may be configured for non-direct interaction, butmay not be configured for direct-interaction. For example, secondrepresentation 1120 may not include any UI elements configured fordirect interaction with a user (e.g., a button, an affordance, auser-interface element, an interactive element, etc.).

In some embodiments, determining to modify the displaying of firstrepresentation 1210 in accordance with a determination that gaze 1250 isdirected to a location that is the same as the location of firstrepresentation 1210 may include a determination that the gaze 1250 hasremained directed to the location that is the same as the location offirst representation 1210 for at least a predetermined period of time,as described with reference to FIGS. 11A and 11B.

As previously described, in embodiments, modifying the firstrepresentation, which may include displaying the second representationmay include animating the modification. For example, the modification ofthe first representation may include animating a change in size of thefirst representation such that the first representation is displayed asgrowing or shrinking, as appropriate, from the current size to the sizeof the second representation. In addition, or in the alternative, themodification of the first representation may include animating the UIelements of the first representation such that the UI elements arepresented as receding into the first representation. In embodiments, theanimation may also include a sound that may be played while theanimation is occurring.

It is noted that, in embodiments, the implementations of the techniquesdescribed herein may include any combination of the features andfunctionalities described above. For example, a representation of avirtual object may be modified to have any one of, and/or anycombination of, a different size, different UI elements, different typesof UI elements (e.g., flat UI elements, protruding UI elements, etc.), adifferent orientation, a different location, a different shape, adifferent brightness, etc.

FIG. 13 is a flow diagram illustrating method 1300 for controlling arepresentation of a virtual object within a CGR environment based oncharacteristics of an input mechanism. In some embodiments, method 1300may be performed by system 100 or a portion of system 100. In someembodiments, method 1300 may be performed by one or more externalsystems and/or devices. In some embodiments, method 1300 may beperformed by system 100 (or a portion of system 100) in conjunction withone or more external systems and/or devices.

At block 1302, the system displays, via a display of an electronicdevice (e.g., a wearable electronic device, an HMD device, etc.), afirst representation of a virtual object within a CGR environment. Forexample, a first representation of a virtual object may be displayed viaa first display (e.g., a left eye display panel) or second display(e.g., a second eye display panel) of an electronic device on arepresentation of a display within the CGR environment. In embodiments,the first representation of the virtual object may be a virtualrepresentation (e.g., a virtual representation superimposed over a firstsurface of the CGR environment via a translucent display of theelectronic device).

In embodiments, the first representation of the virtual object may beconfigured to facilitate non-direct interaction with the virtual object.For example, the first representation of the virtual object may includeat least one UI element of UI elements configured for non-directinteraction such that a user may perceive an interact with the UIelements without directly manipulating the UI elements (e.g., a UIelement configured for output).

In embodiments, the first representation of the virtual object mayinclude at least one UI element of UI elements that may be configurableto facilitate non-direct interaction, but are not configured for directinteraction (e.g., the UI elements may be displayed as protruding 3D UIelements). For example, the UI elements may include a button, anaffordance, a user-interface element, an interactive element, etc.,and/or any combination thereof. When the UI elements are configured tofacilitate direct interaction, a user may select, click, select, and/orotherwise manipulate the UI elements.

In embodiments, a movement of an input mechanism may be detected. Theinput mechanism may include a mechanism configured to facilitateinteraction with the virtual object. For example, the input mechanismmay include a mechanism for a user to manipulate at least one element ofthe representation of the virtual object, or to perceive data providedby the virtual object. In embodiments, the input mechanism may include arepresentation of an appendage of the user (e.g., a finger, hand, leg,foot, etc.), a user's gaze (e.g., head gaze, eye gaze, etc.), an inputdevice (e.g., a mouse, a stylus, etc.), etc. In embodiments, therepresentation of an appendage of the user may include a virtualrepresentation of the appendage and/or may include data representingcharacteristics of the appendage (e.g., location, orientation, distanceto a particular point, etc.) within the CGR environment. In aspects,using input sensors (e.g., touch-sensitive surfaces, image-sensors,etc.) configured to perform hand-tracking, head gaze-tracking, eyegaze-tracking, finger-tracking, etc., a movement of the input mechanismmay be detected. For example, the input mechanism may move from aprevious location to a current location.

In embodiments, in response to the detected movement of the inputmechanism, a determination may be made as to whether the currentlocation of the input mechanism is within the predetermined distancefrom the first representation or not. However, when no movement of theinput mechanism is detected, the determination of whether the currentlocation of the input mechanism is within the predetermined distancefrom the first representation or not may not be performed. In someembodiments, the determination of whether the current location of theinput mechanism is within the predetermined distance from the firstrepresentation or not may be performed when a detected movement isdetermined to be towards the first representation. In these cases, ifthe movement of the input mechanism is determined to be away from thefirst representation, the determination of whether the current locationof the input mechanism is within the predetermined distance from thefirst representation or not may not be performed even though a movementof the input mechanism may be detected.

At block 1304, in accordance with a determination that the currentlocation of the input mechanism is within a predetermined distance fromthe first representation of the virtual object, the system displays, viathe display of the electronic device, a second representation of thevirtual object within the CGR environment. In embodiments, the secondrepresentation of the virtual object may be different from the firstrepresentation of the virtual object.

In embodiments, in response to displaying the second representation ofthe virtual object, the first representation may cease to be displayed.In some embodiments, the second representation may be displayed at thesame location and/or on the same surface where the first representationwas displayed.

In embodiments, the second representation may be configured tofacilitate direct interaction by a user with the associated virtualobject. For example, the second representation may include at least oneUI element of UI elements configured for direct interaction. Inembodiments, the UI elements may include at least one UI elementdisplayed as a flat 2D UI element displayed upon a physical object. Inembodiments, the UI elements may include any one of and/or anycombination of a button, an affordance, a user-interface element, aninteractive element, etc.

In some embodiments, the second representation may have a size that isdifferent than the size of the first representation. For example, thesize of the second representation may be greater than the size of thefirst representation. In embodiments, the second representation mayinclude a portion of the first representation, and the portion of thefirst representation included in the second representation may be largerthan the size of the same portion in the first representation.

In some embodiments, the second representation of the virtual object maybe displayed at a location that is different than the current locationof the first representation. In embodiments, the location at which thesecond representation of the virtual object may be displayed may be alocation that is closer to the user than the current location of thefirst representation. In some embodiments, the second representationdisplayed at the new location may be the same representation as thefirst representation.

In some embodiments, the first representation may be a 3D representationof the virtual object, and the second representation may be a 2Drepresentation of the virtual object. In embodiments, the secondrepresentation may include at least a portion of the virtual object thatis not displayed in the first representation of the virtual object.

As described above, one aspect of the present technology is thegathering and use of data available from various sources to providespecialized resource management of low-power devices with additivedisplays (e.g., HMD devices with additive displays) to conserve batterylife for users and to provide specialized content to users of thelow-power devices. The present disclosure contemplates that, in someinstances, this gathered data may include personal information data thatuniquely identifies or can be used to contact or locate a specificperson. Such personal information data can include demographic data,location-based data, telephone numbers, email addresses, twitter IDs,home addresses, data or records relating to a user's health or level offitness (e.g., vital signs measurements, medication information,exercise information), date of birth, or any other identifying orpersonal information.

The present disclosure recognizes that the use of such personalinformation data, in the present technology, can be used to the benefitof users. For example, the personal information data can be used toconserve battery life of a user's low-power device. Accordingly, forexample, the use of such personal information data the system toproperly manage resources to conserve battery life for the low-powerdevices. Further, other uses for personal information data that benefitthe user are also contemplated by the present disclosure. For instance,health and fitness data may be used to provide insights into a user'sgeneral wellness or may be used as positive feedback to individualsusing technology to pursue wellness goals.

The present disclosure contemplates that the entities responsible forthe collection, analysis, disclosure, transfer, storage, or other use ofsuch personal information data will comply with well-established privacypolicies and/or privacy practices. In particular, such entities shouldimplement and consistently use privacy policies and practices that aregenerally recognized as meeting or exceeding industry or governmentalrequirements for maintaining personal information data private andsecure. Such policies should be easily accessible by users and should beupdated as the collection and/or use of data changes. Personalinformation from users should be collected for legitimate and reasonableuses of the entity and not shared or sold outside of those legitimateuses. Further, such collection/sharing should occur after receiving theinformed consent of the users. Additionally, such entities shouldconsider taking any needed steps for safeguarding and securing access tosuch personal information data and ensuring that others with access tothe personal information data adhere to their privacy policies andprocedures. Further, such entities can subject themselves to evaluationby third parties to certify their adherence to widely accepted privacypolicies and practices. In addition, policies and practices should beadapted for the particular types of personal information data beingcollected and/or accessed and adapted to applicable laws and standards,including jurisdiction-specific considerations. For instance, in the US,collection of or access to certain health data may be governed byfederal and/or state laws, such as the Health Insurance Portability andAccountability Act (HIPAA); whereas health data in other countries maybe subject to other regulations and policies and should be handledaccordingly. Hence different privacy practices should be maintained fordifferent personal data types in each country.

Despite the foregoing, the present disclosure also contemplates examplesin which users selectively block the use of, or access to, personalinformation data. That is, the present disclosure contemplates thathardware and/or software elements can be provided to prevent or blockaccess to such personal information data. For example, in the case ofmanaging resources for low-powered devices, the present technology canbe configured to allow users to select to “opt in” or “opt out” ofparticipation in the collection of personal information data duringregistration for services or anytime thereafter. In another example,users can select not to provide eye tracking data, such as pupillocation, pupil dilation, and/or blink rate for specialized resourcemanagement. In yet another example, users can select to limit the lengthof time the eye-tracking data is maintained or entirely prohibit thedevelopment of a baseline eye tracking profile. In addition to providing“opt in” and “opt out” options, the present disclosure contemplatesproviding notifications relating to the access or use of personalinformation. For instance, a user may be notified upon downloading anapp that their personal information data will be accessed and thenreminded again just before personal information data is accessed by theapp.

Moreover, it is the intent of the present disclosure that personalinformation data should be managed and handled in a way to minimizerisks of unintentional or unauthorized access or use. Risk can beminimized by limiting the collection of data and deleting data once itis no longer needed. In addition, and when applicable, including incertain health related applications, data de-identification can be usedto protect a user's privacy. De-identification may be facilitated, whenappropriate, by removing specific identifiers (e.g., date of birth,etc.), controlling the amount or specificity of data stored (e.g.,collecting location data a city level rather than at an address level),controlling how data is stored (e.g., aggregating data across users),and/or other methods.

Therefore, although the present disclosure broadly covers use ofpersonal information data to implement one or more various disclosedexamples, the present disclosure also contemplates that the variousexamples can also be implemented without the need for accessing suchpersonal information data. That is, the various examples of the presenttechnology are not rendered inoperable due to the lack of all or aportion of such personal information data. For example, resources oflow-powered devices can be managed and content (e.g., status updatesand/or objects) can be selected and delivered to users by inferringpreferences based on non-personal information data or a bare minimumamount of personal information, such as the content being requested bythe device associated with a user, other non-personal informationavailable to the system controlling the low-power device, or publiclyavailable information.

What is claimed is:
 1. A system, comprising: a display; one or moreprocessors; and memory storing one or more programs configured to beexecuted by the one or more processors, the one or more programsincluding instructions for: displaying, via the display, a firstrepresentation of a virtual object at a first location within acomputer-generated reality (CGR) environment, wherein the first locationcorresponds to a first use context of a plurality of use contexts;receiving a request to move the first representation, within the CGRenvironment, to a second location that is different from the firstlocation, wherein the second location corresponds to a location on aplanar surface; and in response to receiving the request: in accordancewith a determination that the second location corresponds to a seconduse context of the plurality of use contexts, displaying, via thedisplay, at the second location, a second representation of the virtualobject based on the second use context, wherein the secondrepresentation is different from the first representation, and whereinthe second representation is a two-dimensional representation; and inaccordance with a determination that the second location corresponds toa third use context of the plurality of use contexts, displaying, viathe display, at the second location, a third representation of thevirtual object based on the third use context, wherein the thirdrepresentation is different from the first representation and the secondrepresentation, and wherein the third representation is athree-dimensional representation.
 2. The system of claim 1, wherein: thesecond use context of the plurality of use contexts includes a usecontext that is satisfied when a determination is made that the secondlocation corresponds to a location on a representation of an electronicdevice; and in accordance with the determination that the secondlocation corresponds to the second use context of the plurality of usecontexts, displaying the second representation of the virtual objectbased on the second use context includes displaying, within the CGRenvironment, the two-dimensional representation on the representation ofthe electronic device.
 3. The system of claim 1, wherein: the second usecontext of the plurality of use contexts includes a use context that issatisfied when a determination is made that the second locationcorresponds to a location on a vertical plane; and in accordance withthe determination that the second location corresponds to the second usecontext of the plurality of use contexts, displaying the secondrepresentation of the virtual object based on the second use contextincludes displaying the two-dimensional representation on the verticalplane within the CGR environment.
 4. The system of claim 3, wherein asize of the second representation is based on a distance between thedisplay and the vertical plane within the CGR environment.
 5. The systemof claim 1, wherein: the third use context of the plurality of usecontexts includes a use context that is satisfied when a determinationis made that the second location corresponds to a location on ahorizontal plane within the CGR environment; and in accordance with thedetermination that the second location corresponds to the third usecontext of the plurality of use contexts, displaying the thirdrepresentation of the virtual object based on the third use contextincludes displaying the three-dimensional representation on thehorizontal plane within the CGR environment.
 6. The system of claim 5,wherein: in accordance with a determination that the horizontal plane isa horizontal plane of a first type, the three-dimensional representationis a representation of a first size; and in accordance with adetermination that the horizontal plane is a horizontal plane of asecond type, the three-dimensional representation is a representation ofa second size that is different from the first size.
 7. The system ofclaim 1, wherein: the second use context of the plurality of usecontexts includes a use context that is satisfied when maximized viewcriteria are satisfied, displaying the second representation of thevirtual object based on the second use context includes displaying aplurality of representations of virtual objects on a plurality of planeswithin the CGR environment.
 8. The system of claim 7, wherein aselectable virtual object that corresponds to a maximized viewaffordance is displayed, and wherein maximized view criteria includes acriterion that is satisfied when the selectable virtual object thatcorresponds to the maximized view affordance is selected.
 9. The systemof claim 1, wherein: the determination is made that the second locationcorresponds to the second use context of the plurality of use contexts;the first representation includes first visual content; and the secondrepresentation does not include the first visual content.
 10. The systemof claim 1, wherein: the determination is made that the second locationcorresponds to the second use context of the plurality of use contexts;the first representation includes third visual content that is displayedat a third size; and the second representation includes the third visualcontent that is displayed at a fourth size that is different from thethird size.
 11. The system of claim 1, wherein: the determination ismade that the second location corresponds to the second use context ofthe plurality of use contexts; the first representation includes a firstselectable object; and the second representation does not include thefirst selectable object.
 12. The system of claim 1, wherein: thedetermination is made that the second location corresponds to the seconduse context of the plurality of use contexts; the first representationis a fourth size; and the second representation is a fifth size that isdifferent from the fourth size.
 13. The system of claim 1, whereindisplaying the second representation of the virtual object based on thesecond use context includes transitioning display of the firstrepresentation to display of the second representation when the firstrepresentation is within a predetermined distance from the secondlocation.
 14. The system of claim 1, wherein the one or more programsfurther include instructions for: in accordance with a determinationthat the second location corresponds to a fourth use context of theplurality of use contexts, wherein the fourth use context is satisfiedwhen the second location corresponds to a prohibited location, forgoingdisplay, within the CGR environment, of a representation of the virtualobject based on the fourth use context.
 15. The system of claim 14,wherein the one or more programs further include instructions for: inaccordance with the determination that the second location correspondsto the fourth use context of the plurality of use contexts, displaying,within the CGR environment, an indication that the second location is aprohibited location.
 16. A non-transitory computer-readable storagemedium storing one or more programs configured to be executed by one ormore processors of a system having a display, the one or more programsincluding instructions for: displaying, via the display, a firstrepresentation of a virtual object at a first location within acomputer-generated reality (CGR) environment, wherein the first locationcorresponds to a first use context of a plurality of use contexts;receiving a request to move the first representation, within the CGRenvironment, to a second location that is different from the firstlocation, wherein the second location corresponds to a location on aplanar surface; and in response to receiving the request: in accordancewith a determination that the second location corresponds to a seconduse context of the plurality of use contexts, displaying, via thedisplay, at the second location, a second representation of the virtualobject based on the second use context, wherein the secondrepresentation is different from the first representation, and whereinthe second representation is a two-dimensional representation; and inaccordance with a determination that the second location corresponds toa third use context of the plurality of use contexts, displaying, viathe display, at the second location, a third representation of thevirtual object based on the third use context, wherein the thirdrepresentation is different from the first representation and the secondrepresentation, and wherein the third representation is athree-dimensional representation.
 17. The non-transitorycomputer-readable storage medium of claim 16, wherein: the second usecontext of the plurality of use contexts includes a use context that issatisfied when a determination is made that the second locationcorresponds to a location on a representation of an electronic device;and in accordance with the determination that the second locationcorresponds to the second use context of the plurality of use contexts,displaying the second representation of the virtual object based on thesecond use context includes displaying, within the CGR environment, thetwo-dimensional representation on the representation of the electronicdevice.
 18. The non-transitory computer-readable storage medium of claim16, wherein: the second use context of the plurality of use contextsincludes a use context that is satisfied when a determination is madethat the second location corresponds to a location on a vertical plane;and in accordance with the determination that the second locationcorresponds to the second use context of the plurality of use contexts,displaying the second representation of the virtual object based on thesecond use context includes displaying the two-dimensionalrepresentation on the vertical plane within the CGR environment.
 19. Thenon-transitory computer-readable storage medium of claim 16, wherein:the third use context of the plurality of use contexts includes a usecontext that is satisfied when a determination is made that the secondlocation corresponds to a location on a horizontal plane within the CGRenvironment; and in accordance with the determination that the secondlocation corresponds to the third use context of the plurality of usecontexts, displaying the second representation of the virtual objectbased on the third use context includes displaying the three-dimensionalrepresentation on the horizontal plane within the CGR environment. 20.The non-transitory computer-readable storage medium of claim 19,wherein: in accordance with a determination that the horizontal plane isa horizontal plane of a first type, the three-dimensional representationis a representation of a first size; and in accordance with adetermination that the horizontal plane is a horizontal plane of asecond type, the three-dimensional representation is a representation ofa second size that is different from the first size.
 21. Thenon-transitory computer-readable storage medium of claim 16, wherein theone or more programs further include instructions for: in accordancewith a determination that the second location corresponds to a fourthuse context of the plurality of use contexts, wherein the fourth usecontext is satisfied when the second location corresponds to aprohibited location, forgoing display, within the CGR environment, of arepresentation of the virtual object based on the fourth use context.22. A method, comprising: displaying, via a display of a wearableelectronic device, a first representation of a virtual object at a firstlocation within a computer-generated reality (CGR) environment, whereinthe first location corresponds to a first use context of a plurality ofuse contexts; receiving a request to move the first representation,within the CGR environment, to a second location that is different fromthe first location, wherein the second location corresponds to alocation on a planar surface; and in response to receiving the request:in accordance with a determination that the second location correspondsto a second use context of the plurality of use contexts, displaying,via the display of the wearable electronic device, at the secondlocation, a second representation of the virtual object based on thesecond use context, wherein the second representation is different fromthe first representation, and wherein the second representation is atwo-dimensional representation; and in accordance with a determinationthat the second location corresponds to a third use context of theplurality of use contexts, displaying, via the display of the wearableelectronic device, at the second location, a third representation of thevirtual object based on the third use context, wherein the thirdrepresentation is different from the first representation and the secondrepresentation, and wherein the third representation is athree-dimensional representation.
 23. The method of claim 22, wherein:the second use context of the plurality of use contexts includes a usecontext that is satisfied when a determination is made that the secondlocation corresponds to a location on a representation of an electronicdevice; and in accordance with the determination that the secondlocation corresponds to the second use context of the plurality of usecontexts, displaying the second representation of the virtual objectbased on the second use context includes displaying, within the CGRenvironment, the two-dimensional representation on the representation ofthe electronic device.
 24. The method of claim 22, wherein: the seconduse context of the plurality of use contexts includes a use context thatis satisfied when a determination is made that the second locationcorresponds to a location on a vertical plane; and in accordance withthe determination that the second location corresponds to the second usecontext of the plurality of use contexts, displaying the secondrepresentation of the virtual object based on the second use contextincludes displaying the two-dimensional representation on the verticalplane within the CGR environment.
 25. The method of claim 22, wherein:the third use context of the plurality of use contexts includes a usecontext that is satisfied when a determination is made that the secondlocation corresponds to a location on a horizontal plane within the CGRenvironment; and in accordance with the determination that the secondlocation corresponds to the third use context of the plurality of usecontexts, displaying the second representation of the virtual objectbased on the third use context includes displaying the three-dimensionalrepresentation on the horizontal plane within the CGR environment.